Demo

Let's use a few examples that show how cluster history helps us understand what is happening in the cluster.

Create the cluster

# First, we'll clean up any previous demo run.
kind delete cluster
rm -rf audit-logs/*
mkdir -p audit-logs
rm -f output.khi
rm -rf data/*

# Then, we'll create a kind cluster with 3 nodes, and enable audit logging on the control plane node.
kind create cluster --config kind-config.yaml

# Now, let's make sure we can access the audit logs.
sudo chmod o+wr audit-logs/kube-apiserver-audit.log

# We'll deploy Kyverno to use with an example.
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace --wait

# And we'll deploy Sloop.
helm install sloop /home/dlipovetsky/projects/sloop/helm/sloop --namespace sloop --create-namespace --wait

# And finally, we'll open the sloop web UI.
kubectl port-forward svc/sloop 8080:80 --namespace sloop

Use Sloop to inspect the cluster history

Open http://localhost:8080 in your browser.

Example 1: How Kubernetes works

In our first example, let's see what happens when we create a Deployment, but the image is not found, so Pods cannot run.

kubectl create deployment nginx --image=not-found --replicas=2

Now let's fix the image, and see what happens!

kubectl set image deploy nginx not-found=nginx

Example 2: What went wrong

In our second example, let's see what happens when we create a Deployment, but a mutating webhook makes the CPU requests and limits invalid, so the Pods cannot run.

First, let's deploy a Kyverno policy that will mutate the Pod spec.

kubectl apply -f kyverno-policy-invalid-cpu-requests-and-limits.yaml

Now let's create a Deployment that will be mutated by the policy.

kubectl create -f sleep-deployment.yaml

And finally, let's delete the policy. We'll on this later.

kubectl delete -f kyverno-policy-invalid-cpu-requests-and-limits.yaml

Use Kubernetes History Inspector (KHI) to process the audit log

Now let's use Kubernetes History Inspector to process the audit log.

# Process the audit log.
/home/dlipovetsky/projects/khi/bin/khi-linux-amd64 \
      --job-mode \
      --job-inspection-type oss-kubernetes-from-files \
      --job-inspection-features ALL \
      --job-inspection-values '{"khi.google.com/oss/form/kube-apiserver-audit-log-files": "/home/dlipovetsky/projects/talk-cluster-history/demo/audit-logs/kube-apiserver-audit.log"}' \
      --job-export-destination ./output.khi

# View the results.
/home/dlipovetsky/projects/khi/bin/khi-linux-amd64 \
    -data-destination-folder /home/dlipovetsky/projects/talk-cluster-history/demo \
    -viewer-mode \
    -viewer-inspection-id output \
    --port 8081

dlipovetsky/talk-kubernetes-cluster-history

README