gravitational / satellite

Simple and extensible monitoring agent / library for Kubernetes: https://gravitational.com/blog/monitoring_kubernetes_satellite/

Satellite

Warning Satellite was archived 2023-07-01.

Please see the Gravity README.md for more information.

Satellite is an agent collecting health information in a Kubernetes cluster. It is both a library and an application. As a library, satellite can be used as a basis for a custom monitoring solution.

Installation

Satellite is included in Gravity. There are currently no binary distributions available.

How to use it

$ satellite help
usage: satellite [<flags>] <command> [<args> ...]

Cluster health monitoring agent

Flags:
  --help   Show help (also see --help-long and --help-man).
  --debug  Enable verbose mode

Commands:
  help [<command>...]
    Show help.

  agent [<flags>]
    Start monitoring agent

  status [<flags>]
    Query cluster status

  version
    Display version


$ satellite agent help
usage: satellite agent [<flags>]

Start monitoring agent

Flags:
  --help                         Show context-sensitive help (also try --help-long and --help-man).
  --debug                        Enable verbose mode
  --rpc-addr=127.0.0.1:7575      List of addresses to bind the RPC listener to (host:port), comma-separated
  --kube-addr="http://127.0.0.1:8080"
                                 Address of the kubernetes API server
  --kubelet-addr="http://127.0.0.1:10248"
                                 Address of the kubelet
  --docker-addr="/var/run/docker.sock"
                                 Path to the docker daemon socket
  --nettest-image="gcr.io/google_containers/nettest:1.8"
                                 Name of the image to use for networking test
  --name=NAME                    Agent name. Must be the same as the name of the local node
  --initial-cluster=INITIAL-CLUSTER  
                                 Initial cluster configuration as a comma-separated list of peers
  --state-dir=STATE-DIR          Directory to store agent-specific state
  --tags=TAGS                    Define a tags as comma-separated list of key:value pairs
  --etcd-servers=http://127.0.0.1:2379
                                 List of etcd endpoints (http://host:port), comma separated
  --etcd-cafile=ETCD-CAFILE      SSL Certificate Authority file used to secure etcd communication
  --etcd-certfile=ETCD-CERTFILE  SSL certificate file used to secure etcd communication
  --etcd-keyfile=ETCD-KEYFILE    SSL key file used to secure etcd communication
  --influxdb-database=INFLUXDB-DATABASE
                                 Database to connect to
  --influxdb-user=INFLUXDB-USER  Username to use for connection
  --influxdb-password=INFLUXDB-PASSWORD
                                 Password to use for connection
  --influxdb-url=INFLUXDB-URL    URL of the InfluxDB endpoint

$ satellite agent --name=my-host --tags=role:master

$ satellite help status
usage: satellite status [<flags>]

Query cluster status

Flags:
  --help           Show context-sensitive help (also try --help-long and --help-man).
  --debug          Enable verbose mode
  --rpc-port=7575  Local agent RPC port
  --pretty         Pretty-print the output
  --local          Query the status of the local node

You can then query the status of the cluster or that of the local node by issuing a status query:

$ satellite status --pretty

resulting in:

{
   "status": "degraded",
   "nodes": [
      {
         "name": "example.domain",
         "member_status": {
            "name": "example.domain",
            "addr": "192.168.178.32:7946",
            "status": "alive",
            "tags": {
               "role": "node"
            }
         },
         "status": "degraded",
         "probes": [
            {
               "checker": "docker",
               "status": "running"
            },
            ...
         ]
      }
   ],
   "timestamp": "2016-03-03T12:19:44.757110373Z",
   "summary": "master node unavailable"
}

Out of the box, the agent requires at least a single master node (agent with role:master). The test will mark the cluster as degraded if no master is available.

Connect the agent to InfluxDB database monitoring:

$ satellite agent --tags=role:master \
	--state-dir=/var/run/satellite \
	--influxdb-database=monitoring \
	--influxdb-url=http://localhost:8086

Building from source

1. Install Go 1.13+
2. Clone the repo: git clone https://github.com/gravitational/satellite.git
3. Enter the repo and run make

Design

Satellites design goals are:

• lightweight periodic testing
• high availability and resilience to network partitions
• no single point of failure
• history of health data as a time series

The health status information is collected in the form of a time series and persisted to a sqlite backend. Additional backends are supported via an interface.

satellite's Issues

Remove dependency on serf

Currently satellite relies on serf as a cluster membership tool to determine which nodes are a part of the cluster, checking their health, etc.

When satellite becomes a standalone tool, it can no longer use serf for that. Instead, it should be using Kubernetes nodes API to retrieve the members of the cluster.

Update README to reflect godep -> go modules migration

Summary

The satellite README.md file mentions that we use 'dep'. However, in #226 we switched from dep to go modules. We should remove the entire dependencies section as it isn't complete or current:

## Dependencies
 - [serf], v0.7.0 or later
 - [godep]

https://github.com/gravitational/satellite/blob/5a67ae767d681f113be3cf5d81ed79bab9d0738c/README.md

Note

I'm filing this as a easy first bug for a git workshop that I'm running.

Request For Translation

Good Day Sir! I want to contribute in translating Filipino language.

Deploy Satellite inside Kubernetes pod

Once dependency on serf is removed (#256), we need to containerize Satellite and deploy it inside a generic Kubernetes cluster.

Things to consider:

• It should run as a DaemonSet.
• Figure out volumes that will need to be mounted into the container.
• Host networking?
• PodSecurityPolicies and RBAC.
• Figure out checks that it can/cannot execute.

Success:

• The design doc contains answers to the above questions, and a sample Satellite DaemonSet spec.

I explored this a little more, and to help add this docs. I started by creating this diagram. Let me know if the gist of it is right? I wasn't 100% clear on how EKS workers work, but wanted to see if I've got the right jist here.

I explored this a little more, and to help add this docs. I started by creating this diagram. Let me know if the gist of it is right? I wasn't 100% clear on how EKS workers work, but wanted to see if I've got the right jist here.

Originally posted by @benarent in gravitational/teleport#2830 (comment)

Create Satellite helm chart

Once we have done research on what it would take to deploy Satellite as a Pod in a generic K8s cluster (#257), create a Helm chart for the Satellite stack.

Success:

• Satellite can be installed in the cluster using helm install command.
• Nethealth is included as a part of the Satellite Helm chart.
  • Double-check whether nethealth can be used on a generic cluster.