This project contains the auto-heal service. It receives alert notifications from the Prometheus alert manager and executes Ansible playbooks to resolve the root cause.
Most of the configuration of the auto-heal service is kept in a YAML
configuration file. The name of the configuration file is specified using the
--config-file command line option. If this option isn't explicitly given then
the service will try to load the autoheal.yml file from the current working
directory.
In addition to the configuration file the auto-heal service also uses command
line options to configure the connection to the Kubernetes API and the log
level. Use the -h option to get a complete list of these command line options.
The --kubeconfig command line option is used to specify the location of the
Kubernetes client configuration file. When running outside of a Kubernetes
cluster the auto-heal service will use $HOME/.kube/config by default, the same
used by the kubectl command. When running inside a Kubernetes cluster it will
use the configuration that Kubernetes mounts automatically in the pod file
system. So in most cases this command line option won't have to be explicitly
included.
Assuming that you want to have your own my.yml configuration file a typical
command line will be the following:
$ autoheal server --config-file=my.yml --logtostderrSee the autoheal.yml file for a complete example.
The first section of the configuration file is named awx and it contains all
the details needed to connect to the AWX
or Ansible Tower server:
awx:
address: https://myawx.example.com/api
proxy: http://myproxy.example.com:3128
credentialsRef:
namespace: my-namespace
name: my-awx-credentials
tlsRef:
namespace: my-namespace
name: my-awx-ca
project: "Auto-heal"The address parameter is the URL of the API of the AWX server. It should
contain the /api suffix, but not the /v1 or /v2 suffix, as the auto-heal
service will internally decide which version to use.
The proxy parameter is optional, and it indicates what HTTP proxy should be
used to connect to the AWX API. If this parameter is not specified, or if it is
empty, then the connection will be direct to the AWX server, without a proxy.
The credentialsRef parameter is a reference to the Kubernetes
secret that contains
the user name and password used to connect to the AWX API. That secret should
contain the username and password keys. For example:
apiVersion: v1
kind: Secret
metadata:
namespace: my-namespace
name: my-awx-credentials
data:
username: YWxlcnQtaGVhbGVy
password: ...Alternatively it is also possible to specify the user name and password directly
inside the configuration file, using the credentials section. For example:
credentials:
username: autoheal
password: ...This is very convenient for development environments, but it is not recommended
for production environments, as then the configuration file needs to be
protected very carefully. For example, you can create a separate file for the
credentials, give it restricted permissions, and then load it using the
--config-file option twice:
$ echo > general.yml <<.
awx:
address: https://myawx.example.com/api
.
$ echo > credentials.yml <<.
credentials:
username: "autoheal"
password: "..."
.
$ chmod u=r,g=,o= credentials.yml
$ autoheal server --config-file=general.yml --config-file=credentials.yml
The tlsRef parameter is a reference to the Kubernetes
secret that contains
the certificates used to connect to the AWX API. That secret should contain the
ca.crt key, for example:
apiVersion: v1
kind: Secret
metadata:
namespace: my-namespace
name: my-awx-tls
data:
ca.crt: |-
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvVENDQWVXZ0F3SUJBZ0lKQUxNRXB6OWxa
VkVzdzI3Sm5BYlMyejNhbUF0YTc1QmNnVGcvOUFCdDV0VVc2VTJOKzkKbXc9PQotLS0tLUVORCBD
...Alternatively it is also possible to specify the CA certificates directly inside
the configuration file, using the tls section. For example:
tls:
caCerts: |-
-----BEGIN CERTIFICATE-----
MIIFgzCCA2ugAwIBAgIPXZONMGc2yAYdGsdUhGkHMA0GCSqGSIb3DQEBCwUAMDsx
CzAJBgNVBAYTAkVTMREwDwYDVQQKDAhGTk1ULVJDTTEZMBcGA1UECwwQQUMgUkFJ
...
-----END CERTIFICATE-----They can also be specified indirectly, putting the name of a PEM file in the
caFile parameter:
tls:
caFile: /etc/autoheal/my-ca.pemThe insecure parameter controls whether to use an insecure connection to the
AWX server. If the connection is insecure then the TLS will not be verified. It
should always be set to false (the default) in production environments.
The project parameter is the name of the AWX project that contains the job
templates that will be used to run the playbooks.
The throttling section of the configuration describes how to throttle the
execution of healing actions. This is intended to prevent healing storms that
could happen if the same alerts are send repeatedly to the service.
The interval parameter controls the time that the service will remember an
executed healing action. If an action is triggered more than once in the given
interval it will be executed only the first time. The rest of the times it will
be logged and ignored. (see autoheal.yml for an example.)
The default interval value is one hour. Leaving the interval parameter 0
will disable throttling altogether.
Note that for throttling purposes actions are considered the same if they have exactly the same fields with exactly the same values after processing them as templates. For example, an action defined like this:
awxJob:
template: "Restart {{ $labels.service }}"Will have different values for the template field if the triggering alerts
have different service labels.
The auto-heal service performs a periodic job status check against AWX server,
to check the status of the active jobs that were triggered.
The jobStatusCheckInterval parameter determines how often to perform this check.
It is optional, and the defult is '5m' (every 5 minutes).
The second important section of the configuration file is rules. It contains
the list of healing rules used by the auto-heal service to decide which action
to run for each received alert. For example:
rules:
- metadata:
name: start-node
labels:
alertname: "NodeDown"
awxJob:
template: "Start node"
extraVars:
node: "{{ $labels.instance }}"
- metadata:
name: start-service
labels:
alertname: ".*Down"
service: ".*"
awxJob:
template: "Start service"The above example contains two healing rules. The first rule will be
executed when the alert received contains a label named alertname with
a value that matches the regular expression NodeDown.
The second rule will be executed when the alert received contains a
labels alertname and service, matching the regular expressions
.*Down and .* respectively.
The metadata parameter of each rule is used to specify the name of
the rule, which is used by the auto-heal service to reference it in log
messages and in metrics.
The labels and annotations parameters of a rule are maps of strings
used to specify the labels and annotations that the alerts should
contain in order to match the rule. The keys of these maps are the names
of the labels or annotations. The values of these maps are regular
expressions that the values of those labels or annotations should match.
The awxJob parameter indicates which job template should be executed
when an alert matches the rule.
The template parameter is the name of the AWX job template.
The extraVars parameter is optional, and if specified it is used to
pass additional variables to the playbook, like with the --extra-vars
option of the ansible-playbook command.
Regardless to the extraVars setting, the content of the alert that
triggered the AWX job will be passed to the playbook as part of
extraVars, in a variable named alert.
The limit parameter is optional, and if specified it is passed to
AWX to constrain the list of hosts managed or affected by the
playbook. Multiple patterns can be separated by colons (:).
As with core Ansible, a:b means "in group a or b", a:b:&c means
"in a or b but must be in c", and a:!b means "in a, and definitely
not in b".
Note that in order to be able to use
extraVarsandlimitmechanisms the AWX job template should have the Prompt on lauch box checked, otherwise the variables passed will be ignored.
The values of all the parameters inside awxJob are processed as Go
templates before executing the
job. These templates receive the details of the alert inside the
$labels and $annotations variables. For example, to generate
dynamically the name of the job templates to execute from the value of
the template annotation of the alert:
awxJob:
template: "{{ $annotations.template }}"Or to pass a variable node to the playbook, calculated from the
instance label:
awxJob:
template: "My template"
extraVars:
node: "{{ $labels.node }}"Limit execution to a host, calculated from the instance label:
awxJob:
template: "My template"
limit: "{{ $labels.instance }}"Follow the upstream Prometheus Alertmanager documentation to configure alerts.
For reference, here is an example Alertmanager configuration that sends an alert to the auto-heal service with authentication. This example assumes autoheal and the Alertmanager are running on the same OpenShift cluster, and requires Alertmanager 0.15 or newer.
global:
resolve_timeout: 1m
route:
group_wait: 1s
group_interval: 1s
repeat_interval: 5m
receiver: autoheal
routes:
- match:
alertname: DeadMansSwitch
repeat_interval: 5m
receiver: autoheal
receivers:
- name: default
- name: deadmansswitch
- name: autoheal
webhook_configs:
- url: https://autoheal.openshift-autoheal.svc/alerts
http_config:
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crtWhen using the cluster-monitoring-operator, save the configuration as
alertmanager.yaml and use this command to apply it:
--namespace=openshift-monitoring \
--from-literal=alertmanager.yaml="$(< alertmanager.yaml)" \
--dry-run -oyaml \
| \
oc replace secret \
--namespace=openshift-monitoring \
--filename=-
To build the binary run this command:
$ make
To build the RPM and the images, run this command:
$ make build-images
To run the automated tests of the project run this command:
$ make check
To manually test the service, without having to have a running Prometheus alert
manager that generates the alert notifications, you can use the *-alert.json
files that are inside the examples directory. For example, to simulate the
NodeDown alert start the server and then use curl to
send the alert notification:
$ autoheal server --config-file=my.yml
$ curl --data @examples/node-down-alert.json http://localhost:9099/alerts
To install the service to an OpenShift cluster use the template contained in
the template.yml file. This template requires at the very minimum the address
and the credentials to connect to the AWX or Ansible Tower server. See the
template.sh script for an example of how to use it.
If needed for development, we can run the server without an OpenShift cluster, simulating OpenShift's alert manager using curl commands.
In the examples dir we have examples of firing alerts, and a configuration file that does not require a connection to a working OpenShift cluster.
To run autoheal in dev mode (without a running OpenShift cluster) developers can use the dev config file in the examples dir.
To simulate alerts firing, developers can use the example alerts.
$ make build
$ make run-dev
$ curl --data @examples/node-down-alert.json http://localhost:9099/alerts
When developing features that does not require AWX server, developers can use a mock-awx server from the examples dir. The mock server will listen on port 8080.
$ cd examples/mock-awx
$ go run mock-awx.go
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent