wjwitek / kube-vip-case-study

Case study of using kube-vip.

kube-vip - Load balancing services for Kubernetes clusters

Ewa Miklewska, Weronika Witek, Bartesz Zbiegień, Jan Ziętek

Table of Contents

1. Introduction
2. Theoretical background
3. Case study concept description
4. Solution architecture
5. Technologies
6. Environment configuration
7. Measuring response time
8. DDOS Test Result
9. Bibliography

1. Introduction

A lot of modern application have to support large number of users and provide reliable service. This is often achieved by hosting multiple copies of the application via frameworks like Kubernetes and having a load balancer. This service is responsible for managing to which copy the traffic goes and allows user to not care about which copy of the app they're connecting to.\ Kube-vip is a framework that provides such load balancing functionality. What differs it from other solutions is that runs directly in the Kubernetes cluster, making it lighter and not reliable on any external software.

2. Theoretical background

Kube-vip is a framework that supports creating highly available Kubernetes clusters. It offers three functionalities: VIP, load balancing in control plane and supporting load balancing of a service. Because of this it always works with multi-node or multi-pod clusters.\ Virtual IP (VIP) allows for ensuring that an endpoint is always available. Within the cluster a leader is elected, which will than assume the VIP. If the leader fails or is changed for other reasons, the VIP is moved to the next leader. When kube-vip is configured to work as a load balancer for control plane it listens for API requests and than distributes them to multiple Kubernete's API servers across many nodes. This is done via TCP-based round-robin protocol on the kernel level (so before packets are sent to real TCP port).\ For the service load balancing, kube-vip takes more of a supporting role. It still requires external load controller, typically connected to the cluster via Cloud Controller. The kube-vip's role in this case is making the load balancer's endpoint visible to the rest of the cluster. For example if a load balancer is taking care of multiple copies of a backend service, kube-vip will advertise the VIP under which they are 'aggregated' to the frontend service in the same cluster.

DOS (denial-of-service) attack is cyber-attack in which the perpetrator seeks to make a machine or network resource unavailable to its intended users by temporarily or indefinitely disrupting services of a host connected to a network. DDOS stands for distributed denial-of-service and it involves attacking via multiple hosts contrary to just one which makes it much harder to detect. Ideally, we would want service to quickly recognize the origin of fake requests and not accept them after recognizing a pattern (static IP address, attacked protocol, specific http request). We can broadly recognize 3 types of DDOS attacks:

• Application Layer attacks - target the layer where web pages are generated on the server and delivered in response to HTTP requests. The basic attack is known as HTTP flood which is similar to pressing refresh in a web browser over and over on many different computers at once – large numbers of HTTP requests flood the server, resulting in denial-of-service.
• Protocol attacks - also known as a state-exhaustion attacks, cause a service disruption by over-consuming server resources and/or the resources of network equipment like firewalls and load balancers by utilizeing weaknesses in layer 3 and layer 4 of the protocol stack to render the target inaccessible. Example of such could SYN Flood that exploits TCP handshake and force service to wait for final step in chandshake that never occurs.
• Volumetric attacks - attempts to create congestion by consuming all available bandwidth between the target and the larger Internet.

In this study we will explore load balancer capabilities against first two types of attacks.

3. Case study concept description

Our case study will explore two main features of Kube-vip: load balancing and Virtual IP. We will perform DDOS attack on a simple cluster connected to network and investigate how kube-vip's load balancer will perform. For VIP testing, we will perform simulation of emergency shut-down of selected control nodes to check how long it takes to recover server functionality. Diagram below presents architecture of the case study concept.

4. Solution architecture

Planned architecture includes 3 server nodes and two agent nodes.

5. Technologies

• Kube-vip - load balancing framework for Kubernetes cluster.
• Docker - creating containers managed by Kubernetes
• HULK - Http Unbearable Load King - web server denial of service testing tool,
• hping3 - network tool able to send custom ICMP/UDP/TCP packets and to display target replies. It will be used for more advanced DOS tests.

6. Environment configuration

We are setting up our enironment using KinD. Kube-vip is set up as a DeamonSet and also serves as Cloud Controller for our cluster. Here are the steps:

1. Creat cluster using a config file

kind create cluster --config ./manifests/kind-config.yaml

2. Get address pool for kube-vip

docker network inspect kind -f '{{ range $i, $a := .IPAM.Config }}{{ println .Subnet }}{{ end }}'

The result should be a subnet like: 172.18.0.0/16 3. Deploy the kube-vip Cloud Controller

kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml

4. Add address range belonging to subnet we got in 2.

kubectl create configmap --namespace kube-system kubevip --from-literal range-global=172.18.100.10-172.18.100.30

5. Create kube-vip's RBAC settings

kubectl apply -f https://kube-vip.io/manifests/rbac.yaml

6. Export environment variables

export KVVERSION=latest
export INTERFACE={interface used by nodes, commonly eth0}
export VIP={our chosen VIP, must be in subnet from point 2, e.g. 172.18.0.10}

7. Prepare kube-vip command

alias kube-vip="ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION; ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"

8. Generate and apply kube-vip's manifest

kube-vip manifest daemonset --interface $INTERFACE
                            --address $VIP
                            --inCluster
                            --taint
                            --controlplane
                            --services
                            --arp
                            --enableLoadBalancer | kubectl apply -f -

9. Replace addres in kubectl with VIP

kubectl config set-cluster kind-kind --server=https://$VIP:6443

Now the kube-vip is working and we should be able to connect to control plane using VIP address, however we need to use flag --insecure-skip-tls-verify. To add the VIP to cluster s certificates additional steps are needed.

1.Get current kubeadm config

kubectl -n kube-system get configmap kubeadm-config --insecure-skip-tls-verify -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

2.Modify the kubeadm.yaml by adding VIP under certSANs, file beginning should look similar to this:

apiServer:
  certSANs:
  - localhost
  - 127.0.0.1
  - 172.20.0.10
  extraArgs:
    runtime-config: ""
  timeoutForControlPlane: 4m0s

3. Connect to a chosen node that hosts a control plane

docker exec -it {container id} bash

4. Move the old certificates

mv /etc/kubernetes/pki/apiserver.{crt,key} ~

5. Get the kubeadm.yaml file created earlier inside the docker container.
6. Generate new certificate

kubeadm init phase certs apiserver --config kubeadm.yaml

7. Repeat the process on other nodes with control planes, howevert instead of creating new certificates copy /etc/kubernetes/pki/apiserver.{crt,key} from the first container.
8. On each container run

kubeadm init phase upload-config kubeadm --config kubeadm.yaml

Now you should be able to use kubectl without the flag --insecure-skip-tls-verify.

7. DDOS Stress Test

For Protocol Attack we will use docker image of Hping3 packet generator image from dockerhub.

docker pull sflow/hping3

We will carry out syn flood attack (-S flag) to our server (-p 80) by sending 10.000 packets of 120B size and randomizing our source to avoid detection:

docker run --rm sflow/hping3 -c 10000 -d 120 -S -w 64 --rand-source <ip address>

8. Measuring response time

We prepared a script to measure average response time of cluster. Running it requires small changes, to allow proper authentication. First we need to add an appropriate role:

1. Create role

kubectl apply -f ./manifests/cluster-role.yaml

2. Create role binding

kubectl create clusterrolebinding service-reader-pod --clusterrole=service-reader --serviceaccount=default:default

Then we need to generate a bearer token:

1. Generate token

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: default-token
  annotations:
    kubernetes.io/service-account.name: default
type: kubernetes.io/service-account-token
EOF

2. Get the token

kubectl get secret default-token -o jsonpath='{.data.token}' | base64 --decode

To use the scrip type:

python3 {api_url} {num_requests_per_batch} "Bearer {token}"

9. DDOS Test Result

The experiment was conducted by measuring average response time for http request. We wanted to see how low scale denial of service attack will increase response time. The requests are done asynchronously in batches of 100 and 2000000 requests are done in total.

On cluster without configures Kube-vip we notice quite visible increase in response time (table below), but not exactly significant because of only one source of attack.

Standard response time	DDOS response time	% increase
0.00525	0.00573	9.1%

On cluster with configured Kube-vip we observe equal standard resolve time, but much bigger increase in repose time during SYN flood attack.

Standard response time	DDOS response time	% increase
0.00538	0.00651	21.0%

We are not exactly sure why Kube-vip so drastically increase respond time during attack.

10. Bibliography

1. The Linux Foundation. Dokumentacja kube-vip. https://kube-vip.io/
2. CNCF. Dokumentacja Kubernetes. https://kubernetes.io/
3. The Kubernetes Authors. Dokumentacja kind. https://kind.sigs.k8s.io/