isabella232 / sensu-kubernetes-events

Sensu Kubernetes Events Check

Table of Contents

• Discussion
• Overview
• Usage examples
  • API Authentication
  • Namespaces
  • Object kind
  • Event types
  • Label selectors
  • Status map
• Configuration
  • Asset registration
  • Check definition
• Installation from source
• Additional notes
• Contributing

Discussion

This plugin is in its early stages of development and we welcome your feedback on it and other future Kubernetes plugins. Please visit the Kubernetes SIG on the Sensu Community Forums to provide feedback and submit feature requests.

Overview

The Sensu Kubernetes events check is a Sensu Check that uses the Kubernetes Event API to identify events that should generate corresponding Sensu events.

This check should be thought of as a meta-check. The check itself, unless it encounters issues (e.g. trouble authenticating with Kubernetes for API access), will always return an OK status (exit code 0). However, for each matching event type it does find, it will create separate events using the agent API.

Given the above, when a matching event occurs, the check will need to be able to connect to the agent API on http://127.0.0.1:3031/events.

Usage examples

Sensu Kubernetes events check

Usage:
  sensu-kubernetes-events [flags]
  sensu-kubernetes-events [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -a, --agent-api-url string     The URL for the Agent API used to send events (default "http://127.0.0.1:3031/events")
  -t, --event-type string        Query for fieldSelector type (supports = and !=) (default "!=Normal")
  -e, --external                 Connect to cluster externally (using kubeconfig)
  -h, --help                     help for sensu-kubernetes-events
  -c, --kubeconfig string        Path to the kubeconfig file (default $HOME/.kube/config)
  -l, --label-selectors string   Query for labelSelectors (e.g. release=stable,environment=qa)
  -n, --namespace string         Namespace to which to limit this check (defaults to check's namespace, use "all" for all namespaces)
  -k, --object-kind string       Object kind to limit query to (Pod, Cluster, etc.)
  -s, --status-map string        Map Kubernetes event type to Sensu event status (default "{\"normal\": 0, \"warning\": 1, \"default\": 3}")

Use "sensu-kubernetes-events [command] --help" for more information about a command.

Namespaces

By default this check assumes your Sensu namespace matches up with your Kubernetes namespace and therefore uses that same namespace when querying the API for events. You can override this with the --namespace flag. To have one check run for events from all Kubernetes namespaces, you can specify --namespace all.

API authentication

In order to query the API, the check must authenticate. The normal use case would be for the check to be running in a container in a Kubernetes pod and would make use of the rest.InClusterConfig() function to handle API host discovery and authentication automatically. That is described here. This is the default behavior.

To use "external" access requires the use of kubeconfig files similar to the kubectl command. This method is enabled via the --external flag. Additionally, the --kubeconfig option can be used to point to an alternative kubeconfig file.

Object kind

If an object kind is not specified via the --object-kind argument, events for all object kinds (cluster, pod, etc.) will be returned.

Event types

The expected use case for this check is to find anomalous events in your Kubernetes environment(s). For that reason, the default event type is !=Normal.

Label selectors

Label selectors can be used to limit the scope of the Kubernetes events returned and checked against the requested event type. You can specify multiple selectors by separating them with commas as the value for the --label-selectors argument.

Status map

The status map allows you to map the event type (e.g. Normal, Warning) to a Sensu event check result. It is a simple JSON map represented as a string. The event types are case-insensitive. The default, below, shows that Normal maps to OK (0), Warning maps to Warning (1), and Default (anything else) maps to Unknown (3):

{
  "Normal": 0,
  "Warning": 1,
  "Default": 3
}

Configuration

Asset registration

Sensu Assets are the best way to make use of this plugin. If you're not using an asset, please consider doing so! If you're using sensuctl 5.13 with Sensu Backend 5.13 or later, you can use the following command to add the asset:

sensuctl asset add sensu/sensu-kubernetes-events

If you're using an earlier version of sensuctl, you can find the asset on the Bonsai Asset Index.

Check definition

---
type: CheckConfig
api_version: core/v2
metadata:
  name: sensu-kubernetes-events
  namespace: default
spec:
  command: >-
    sensu-kubernetes-events
    --agent-api-url http://127.0.0.1:3031/events
    --event-type "!=Normal"
  subscriptions:
  - system
  runtime_assets:
  - sensu/sensu-kubernetes-events
  stdin: true
  handlers:
  - slack

Notes:

• The check definition requires stdin be set to true.
• Any Events created by this check will include the handlers defined for it.

Installation from source

The preferred way of installing and deploying this plugin is to use it as an Asset. If you would like to compile and install the plugin from source or contribute to it, download the latest version or create an executable binary from this source.

From the local path of the sensu-kubernetes-events repository:

go build

Additional notes

Contributing

For more information about contributing to this plugin, see Contributing.