kandros / k8s_webhook_helloworld

Kubernetes Auth/Authz WebHook Hellworld sample

Kubernetes WebHook Authentication/Authorization Minikube HelloWorld

Sample minimal HelloWorld application for Miikube demonstrating Kubernetes Authentication/Authorization using Webhooks.

Webhooks proivde a mechanism for delegating k8s AU/AZ decisions. In the case here, both policy decisions are delegated to an external HTTP REST service. For more information on WebHooks:

• WebHook Authentication
• WebHook Authorization

This repo is designed to run onlocally with minikube while the WebHook server may run locally or remotely as a separate python Flask applicaton (eg Appengine). The steps detailed below are lengthy and involve copying certificates into your minikube' persistent volume and running your webhook server.

You can deploy the AuthN/AuthZ server as a service within the k8s cluster and provide the Cluster DNS entry reference to it from the webhook configuration files. More information about that configuration in the appendix.

Note: This is just a sample helloworld app. Do not use this in production!

---

Installation

Clone the repository

git clone salrashid123/k8s_webook_helloworld

Deploy WebHook Server

It is MUCH easier to deploy your webhook server to AppEngine than to run locally. The latter requires you to play some games with the /etc/hosts file for name resolution and to match the SAN values within the server certificate. Running the webhook server locally is described in the Appendix.

Deploy to Google Appengine

Set up a google cloud platform project and enable AppeEngine

sudo apt-get update
sudo apt-get install python-pip openssl -y
sudo pip install virtualenv

cd k8s_webhook_helloworld/server

pip install -t lib -r requirements.txt

now deploy the webhook server:

gcloud app deploy --version 1 --no-promote

At this point, your Webhook server should be accessible at:

gcloud config  get-value core/project

curl https://1-dot-webhook-dot-YOUR_PROJECT.appspot.com/

Prepare Minikube configuration for Webhook

Minikube needs to know the authn/authz config files and CA webhook_plugin certs to trust but those need to be accesible while minikube is started. That is, the config, certs and key needs to exist within the minikube VM while its starting up. The easiest way to do this is to use host mount folder and reference the files on startup. However, not all minikube drivers support mount folders (as was the case for me; i used -vm-driver kvm2. One workaround employed for kvm2 (and described in the appendix), is to first start minikube, then create the certificate files in a peristent volume folder (eg /var/lib/localkube/

Start Minikube without custom configuration

$ minikube start

$ minikube ssh

$ sudo su -

Specify authn/authz definitions

Create the following files with the values shown below and download the certificats.

For convenience, the default template files are provided here

curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/authn.yaml -o /var/lib/localkube/authn.yaml
curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/authz.yaml -o /var/lib/localkube/authz.yaml
curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/CA/GAE_CA.pem -o /var/lib/localkube/certs/webhook_ca.crt
curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/CA/webhook_plugin.crt -o /var/lib/localkube/certs/webhook_plugin.crt
curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/CA/webhook_plugin.key -o /var/lib/localkube/certs/webhook_plugin.key

Note, we are downloading GAE_CA.pem as the trusted CA. You can find those values for the CA here:

nslookup 1-dot-webhook-dot-YOUR_PROJECT.appspot.com

echo | openssl s_client -showcerts -servername 1-dot-webhook-dot-YOUR_PROJECT.appspot.com -connect 172.217.6.84:443 2>/dev/null

Edit the following files while within minikube

• /var/lib/localkube/authn.yaml
• /var/lib/localkube/authz.yaml

Change the server: line to match your deployed GAE server: (ofcourse substitute YOUR_PROJECT with the value for your GCP project)

My project is mineral-minutia-820 so I will use:

server: https://1-dot-webhook-dot-mineral-minutia-820.appspot.com/authenticate

• authn.yaml

clusters:
  - name: my-authn-service
    cluster:
      certificate-authority: /var/lib/localkube/certs/webhook_ca.crt
      server: https://1-dot-webhook-dot-YOUR_PROJECT.appspot.com/authenticate

users:
  - name: my-api-server
    user:
      client-certificate: /var/lib/localkube/certs/webhook_plugin.crt
      client-key: /var/lib/localkube/certs/webhook_plugin.key

current-context: webhook
contexts:
- context:
    cluster: my-authn-service
    user: my-api-sever
  name: webhook

• authz.yaml

clusters:
  - name: my-authz-service
    cluster:
      certificate-authority: /var/lib/localkube/certs/webhook_ca.crt
      server: https://1-dot-webhook-dot-YOUR_PROJECT.appspot.com/authorize

users:
  - name: my-api-server
    user:
      client-certificate: /var/lib/localkube/certs/webhook_plugin.crt
      client-key: /var/lib/localkube/certs/webhook_plugin.key

current-context: webhook
contexts:
- context:
    cluster: my-authz-service
    user: my-api-sever
  name: webhook

Stop Minikube

minikube stop

Since the config and cert files are saved under /var/lib/localkube/, they will remain after minikube restarts

Restart Minikube with webhook

Now Restart Minikube with the webhook configurations

$ minikube start --extra-config apiserver.Authentication.WebHook.ConfigFile=/var/lib/localkube/authn.yaml \
    --extra-config apiserver.Authorization.Mode=Webhook \
    --extra-config apiserver.Authorization.WebhookConfigFile=/var/lib/localkube/authz.yaml

You can verify connectivity from minikube --> webhook server by entering minikube and testing the endpoint:

Make sure you can connect from the VM to the webhook server by name:

minikube ssh

curl -vk https://1-dot-webhook-dot-YOUR_PROJECT.appspot.com/

Verify Webhook callback

Minikube is now configured to talk to the local webhook server. Verification can be done by either directly invoking the k8s API server or via kubectl.

Calling k8s with Bearer Token directly

First step is we need to know the endpoint of the minikube-k8s sever:

to do that simply run

$ minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.39.196

Then invoke the API server with the k8s server endpoint IP and token:

curl -vk \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InVzZXIxQGRvbWFpbi5jb20ifQ.W0Ek34LU4WQOxXdTqZ9Z-0kESz0wIEdYehxZHlTjt2I" \
  https://192.168.39.196:8443/api/

All endpoints are authorized for the token above. The only endpoint that isn't is /api/pods

The token above can be anything becuase the sample Webhook approves all requests for /authenticate and /authorize

You can verify by inspecting the logs on AppEngine:

Calling k8s with kubectl

If you want to use kubectl, you need to configure a context that will use either a token or basic auth:

For example, the following ~/.kube/config sets up two user contexts that you can use (webhook1 and webhookd2)

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/srashid/.minikube/ca.crt
    server: https://192.168.39.196:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
  
- context:
    cluster: minikube
    user: user1
  name: webhook1

- context:
    cluster: minikube
    user: user2
  name: webhook2

current-context: webhook1
kind: Config
preferences: {}
users:
- name: minikube
  user:
    as-user-extra: {}
    client-certificate: /home/srashid/.minikube/client.crt
    client-key: /home/srashid/.minikube/client.key
- name: user1
  user:
    token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InVzZXIxQGRvbWFpbi5jb20ifQ.W0Ek34LU4WQOxXdTqZ9Z-0kESz0wIEdYehxZHlTjt2I
- name: user2
  user:
    token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InVzZXIyQGRvbWFpbi5jb20ifQ.DTvRw2dBVBOxOyt-Osq2e0iblh_xcbEy-Ir0ZBkkSdY

Swap contexts:

$ kubectl config use-context  webhook1
Switched to context "webhook1".

Verify you can still access cluter informaton

$ kubectl get no
NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     <none>    2d        v1.9.0

Either way, you should see the Authentication and Authorization requests in the window where your webhook server is running:

Authentication Request

{
    "apiVersion": "authentication.k8s.io/v1beta1", 
    "kind": "TokenReview", 
    "metadata": {
        "creationTimestamp": null
    }, 
    "spec": {
        "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InVzZXIxQGRvbWFpbi5jb20ifQ.W0Ek34LU4WQOxXdTqZ9Z-0kESz0wIEdYehxZHlTjt2I"
    }, 
    "status": {
        "user": {}
    }
}

Authenticaiton Response

{
    "apiVersion": "authentication.k8s.io/v1beta1", 
    "kind": "TokenReview", 
    "status": {
        "authenticated": true, 
        "user": {
            "extra": {
                "extrafield1": [
                    "extravalue1", 
                    "extravalue2"
                ]
            }, 
            "groups": [
                "developers", 
                "qa"
            ], 
            "uid": "42", 
            "username": "[email protected]"
        }
    }
}

Authoriztion Request

{
    "apiVersion": "authorization.k8s.io/v1beta1", 
    "kind": "SubjectAccessReview", 
    "metadata": {
        "creationTimestamp": null
    }, 
    "spec": {
        "extra": {
            "extrafield1": [
                "extravalue1", 
                "extravalue2"
            ]
        }, 
        "group": [
            "developers", 
            "qa", 
            "system:authenticated"
        ], 
        "nonResourceAttributes": {
            "path": "/api/", 
            "verb": "get"
        }, 
        "uid": "42", 
        "user": "[email protected]"
    }, 
    "status": {
        "allowed": false
    }
}

Authoriztion Response

{
    "apiVersion": "authorization.k8s.io/v1beta1", 
    "kind": "SubjectAccessReview", 
    "status": {
        "allowed": true
    }
}

Appendix

The following section is optional and details how you can override the certificates to create your own CA and populate the server's certificate with a CN and SAN of your choosing:

Create webhook server certificates

• Create your own CA:

 openssl genrsa -out CA_key.pem 2048
 
 openssl req -x509 -new -nodes -key CA_key.pem -out CA_crt.pem -config openssl.cnf \
   -subj "/C=US/ST=California/L=Mountain View/O=Google/OU=Enterprise/CN=MyCA"

• Edit openssl.cnf and add in the DNS/IP you want to use

cd k8s_webook_helloworld/certs
vi openssl.cnf

set the SAN values:

[alt_names]
#IP.1 = 35.202.144.185
DNS.1 = webhook.domain.local
DNS.2 = webhook-srv
DNS.3 = webhook-srv.kube-system
DNS.4 = webhook-srv.kube-system.svc
DNS.5 = webhook-srv.kube-system.svc.cluster.local

Note: DNS2->5 are there to support having webhook server hosted as a k8s service (webhook-svc)

• Create the server certificate and specify the CN= inline

openssl genrsa -out server.key 2048

openssl req -config openssl.cnf -out server.csr -key server.key -new -sha256 \
   -subj "/C=US/ST=California/L=Mountain View/O=Google/OU=Enterprise/CN=webhook.domain.local"

openssl ca -config openssl.cnf -days 365 -notext  -in server.csr   -out server.crt

Check the certificate created has the corect CN and SAN values:

openssl x509 -in server.crt  -text -noout

see:

    CN = webhook.domain.local

        X509v3 extensions:
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Alternative Name: 
                DNS:webhook.domain.local, DNS:webhook-srv, DNS:webhook-srv.kube-system, DNS:webhook-srv.kube-system.svc, DNS:webhook-srv.kube-system.svc.cluster.local
            X509v3 Key Usage: 
                Digital Signature, Non Repudiation, Key Encipherment

• Generate the plugin certificate

openssl genrsa -out webhook_plugin.key 2048
openssl req -config openssl.cnf -out webhook_plugin.csr -key webhook_plugin.key -new -sha256 \
   -subj "/C=US/ST=California/L=Mountain View/O=Google/OU=Enterprise/CN=webhook_plugin.default.cluster.local"

openssl ca -config openssl.cnf -days 365 -notext  -in webhook_plugin.csr   -out webhook_plugin.crt

IP address for SAN

You can also use an IP address as the endpoint if you do not have an external DNS server available.

To use an IP instead, change the SAN value to include it and regenerate the server.crt/key

edit CA/openssl.cnf and add IP.1 value

[alt_names]
IP.1  = 35.202.144.185
DNS.1 = webhook.domain.local
DNS.2 = webhook-srv
DNS.3 = webhook-srv.kube-system
DNS.4 = webhook-srv.kube-system.svc
DNS.5 = webhook-srv.kube-system.svc.cluster.local

You should see:

X509v3 Subject Alternative Name: 
    IP Address:35.202.144.185, DNS:webhook-srv, DNS:webhook-srv.kube-system, DNS:webhook-srv.kube-system.svc, DNS:webhook-srv.kube-system.svc.cluster.local

Creating new users

The sample here used HMAC JWT to create and decode the tokens:

import jwt
encoded = jwt.encode({'username': '[email protected]'}, 'secret', algorithm='HS256')
print encoded

decoded = jwt.decode(encoded, 'secret', algorithms=['HS256'])
print decoded

Running WebHook locally

As mentioned, getting the webhook server running locally is a bit challenging since minikube needs to connect to the local server and match the CN=, SAN values.

If you would like to attempt this:

Find External interface IP address for your workstation

On your workstation, find its external interface's IP address

/sbin/ifconfig -a
wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.17.134.45  netmask 255.255.224.0  broadcast 10.17.159.255

In the example above, the ip is 10.17.134.45

Minikube uses the host interface so using this, minkube can find the webhook server running on the workstation

Specify CA trust certificate and plugin certs

NOTE: specifying the CA and certficates here properly is critical!

Create the following files:

• /var/lib/localkube/certs/webhook_ca.crt
  • default: webhook_ca.crt
• /var/lib/localkube/certs/webhook_plugin.crt
  • default: webhook_plugin.crt
• /var/lib/localkube/certs/webhook_plugin.key
  • default: webhook_plugin.key

You can use the default files provided in this repo

curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/authn.yaml -o /var/lib/localkube/authn.yaml
curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/authz.yaml -o /var/lib/localkube/authz.yaml
curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/CA/CA_crt.pem -o /var/lib/localkube/certs/webhook_ca.crt
curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/CA/webhook_plugin.crt -o /var/lib/localkube/certs/webhook_plugin.crt
curl -s  https://raw.githubusercontent.com/salrashid123/k8s_webhook_helloworld/master/CA/webhook_plugin.key -o /var/lib/localkube/certs/webhook_plugin.key

Note the CA_crt.pem that is downloaded in the step above is for the self-signed CA included in this repo

Stop Minikube

minikube stop

Restart minikube specifying WebHook

$ minikube start --extra-config apiserver.Authentication.WebHook.ConfigFile=/var/lib/localkube/authn.yaml \
    --extra-config apiserver.Authorization.Mode=Webhook \
    --extra-config apiserver.Authorization.WebhookConfigFile=/var/lib/localkube/authz.yaml

Start WebHook Server

Now run the webhook server:

cd server
virtualenv env
source env/bin/activate
pip install -r requirements

Start the server:

python webhook.py

The default certificates for the webhook server in this git repo uses

• CN: CN=webhook.domain.local

• SAN: X509v3 Subject Alternative Name: DNS: webhook.domain.local

You can change the CN and SAN specifications as well as define your own custom CA. Instructions for that can be found in the Appendix.

SSH to minikube and set the hosts file

minikube ssh

sudo su -

vi /etc/hosts

10.17.134.45  webhook.domain.local

(the value 10.17.134.45 will be different for you and is the public IP we acquired earlier)

Verify minikube->local webhook connectivity

Check if you have local connectivity

curl -vk https://webhook.domain.local:8081/

If that works, you're all set to run minikube w/ webhook locally

Troubleshooting

• Verify connectivity from minikube VM to your AU/AZ server:

minikube ssh

then from within minikube, check connectivity:

$ curl -vk https://webhook.domain.com:8081/

If that fails, then there maybe several reasons:

• Local firewall/iptables If you have iptabbles running, it may interfere with vm->host routing. Please disable iptable rules if AU/AZ is running on the local workstation

• Check /etc/hosts value within the VM and set it to the external IP address of your workstaiton

• Provider--vm-driver kvm2

References

• https://github.com/pasientskyhosting/kubernetes-authserver