doytsujin / kubernetes-oom-event-generator

Generate a Kubernetes Event when a Pod's container has been OOMKilled

kubernetes-oom-event-generator

Generates Kubernetes Event when a container is starting and indicates that it was previously out-of-memory killed.

Design

The Controller listens to the Kubernetes API for new Events and changes to Events. Every time a notification regarding an Event is received it checks whether this Event refers to a "ContainerStarted" event, based on the Reason for the Event and the Kind of the involved object. If this is the case and the Event constitutes a change (meaning it is not a not-changing update, which happens when the resync, that is executed every two minutes, is run) it checks the underlying Pod resource. Should the LastTerminationState of the Pod refer to an OOM kill the controller will emit a Kubernetes Event with a level of Warning and a reason of PreviousContainerWasOOMKilled.

Usage

Usage:
  kubernetes-oom-event-generator [OPTIONS]

Application Options:
  -v, --verbose= Show verbose debug information [$VERBOSE]
      --version  Print version information

Help Options:
  -h, --help     Show this help message

Run the pre-built image xingse/kubernetes-oom-event-generator locally (with local permission):

echo VERBOSE=2 >> .env
docker run --env-file=.env -v $HOME/.kube/config:/root/.kube/config xingse/kubernetes-oom-event-generator

Deployment

Example Clusterrole:

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: xing:controller:kubernetes-oom-event-generator
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/status
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
      - list
      - watch

Run this controller on Kubernetes with the following commands:

kubectl create serviceaccount kubernetes-oom-event-generator \
  --namespace=kube-system

kubectl create -f path/to/example-clusterrole.yml
# alternatively run: `cat | kubectl create -f -` and paste the above example, hit Ctrl+D afterwards.

kubectl create clusterrolebinding xing:controller:kubernetes-oom-event-generator \
  --clusterrole=xing:controller:kubernetes-oom-event-generator \
  --serviceaccount=kube-system:kubernetes-oom-event-generator

kubectl run kubernetes-oom-event-generator \
  --image=xingse/kubernetes-oom-event-generator \
  --env=VERBOSE=2 \
  --serviceaccount=kubernetes-oom-event-generator \
  --namespace=kube-system

Alerting on OOM killed pods

There are many different ways to send alerts when an OOM occurs. We just want to mention two of them here.

Forwarding OOM events to Graylog

Graylog is a popular log management solution, and it includes an alerting feature. See the Graylog docs for more details.

At XING we forward all Kubernetes cluster events to Graylog using our kubernetes-event-forwarder-gelf. This allows us to configure alerts whenever a PreviousContainerWasOOMKilled event generated by the kubernetes-oom-event-generator occurs.

Using kube-state-metrics and Prometheus alerts

When kube-state-metrics is deployed in the cluster and a Prometheus installation is scraping the metrics, you can alert on OOM-killed pods using the prometheus alert manager.

Example alert:

alert: ComponentOutOfMemory
expr: sum_over_time(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[5m])
  > 0
for: 10s
labels:
  severity: warning
annotations:
  description: Critical Pod {{$labels.namespace}}/{{$labels.pod}} was OOMKilled.

The downside is that kube_pod_container_status_terminated_reason always returns to 0 once a container starts back up. See the introduction of kube_pod_container_status_last_terminated_reason for more details.

Developing

You will need a working Go installation (1.11+) and the make program. You will also need to clone the project to a place outside you normal go code hierarchy (usually ~/go), as it uses the new Go module system.

All build and install steps are managed in the central Makefile. make test will fetch external dependencies, compile the code and run the tests. If all goes well, hack along and submit a pull request. You might need to modify the go.mod to specify desired constraints on dependencies.

Make sure to run go mod tidy before you check in after changing dependencies in any way.

Releases

Releases are a two-step process, beginning with a manual step:

• Create a release commit
  • Increase the version number in kubernetes-oom-event-generator.go/VERSION
  • Adjust the CHANGELOG
• Run make release, which will create an image, retrieve the version from the binary, create a git tag and push both your commit and the tag

The Travis CI run will then realize that the current tag refers to the current master commit and will tag the built docker image accordingly.