kubernetes-homelab

A repository to keep resources and configuration files used with my Kubernetes homelab.

Content of the Repository

alertmanager - configuration files to deploy Alertmanager.
ansible - Ansible playbooks to deploy Kubernetes homelab.
calico - configuration files to deploy Calico CNI.
cka - CKA study notes.
dashboard - configuration files to deploy Kubernetes dashboard.
docs - images and documentation files.
grafana - configuration files to deploy Grafana.
httpd-healthcheck - configuration files deploy a simple httpd healthcheck for Istio ingressgateway.
istio - configuration files to deploy Istio.
istio-addons - configuration files to deploy Istio add-ons (e.g. Kiali).
kube-state-metrics - configuration files to deploy kube-state-metrics.
kubecost - configuration files to deploy Kubecost.
logging - configuration files to deploy Elastic Stack (Elasticsearch, Kibana etc).
metallb - configuration files to deploy MetalLB.
mikrotik-exporter - configuration files to deploy a Prometheus exporter for Mikrotik devices.
openvpn - configuration files to deploy OpenVPN server.
packer - configuration files build Qemu/KVM images with Packer.
pihole-exporter - configuration files to deploy a Prometheus exporter for Pi-hole Raspberry Pi ad blocker.
pii-demo - a demo PII application based on Apache, PHP and MySQL to test Istio's mTLS.
pii-demo-blue-green - a demo PII application based that uses blue/green deployment.
prometheus - configuration files to deploy Prometheus monitoring.
pxe - configuration files for PXE boot and Kickstart.
regcred - docker registry credentials.
speedtest-influxdb - configuration files to deploy a Speedtest service that stores results in InfluxDB.
terraform - configuration files to manage Kubernetes with Terraform.
truenas-nfs - configuration files to deploy democratic-csi with TrueNAS NFS.
x509-certificate-exporter - configuration files to deploy x509-certificate-exporter.

Homelab Network Diagram

Network Configuration

Network is configured as follows:

LAN: 10.11.1.0/24.
Gateway: 10.11.1.1.
DNS/DHCP/NTP/SMTP servers: 10.11.1.2 and 10.11.1.3.
Managed switch: 10.11.1.4 currently no special config but a couple of VLANs to separate homelab devices from the rest of the home network.
PXE boot server: 10.11.1.20.
DNS private zone: hl.test (a reserved top level DNS name .test, see rfc2606).
DHCP: range 10.11.1.140-10.11.1.149.

Hostnames and their IP addresses:

Hostname	IP Address	Information
mikrotik.hl.test	10.11.1.1	Mikrotik router
admin1.hl.test	10.11.1.2	DNS/DHCP master, NTP, SMTP, HAProxy master, Keepalived
admin2.hl.test	10.11.1.3	DNS/DHCP slave, NTP, SMTP, HAProxy backup, Keepalived
switch.hl.test	10.11.1.4	Managed switch
truenas.hl.test	10.11.1.5	TrueNAS Core shared storage server for Kubernetes
pi.hl.test	10.11.1.7	RaspberryPi Pi-hole DNS ad blocker
kvm1.hl.test	10.11.1.21	KVM hypervisor
kvm2.hl.test	10.11.1.22	KVM hypervisor
kvm3.hl.test	10.11.1.23	KVM hypervisor
kubelb.hl.test	10.11.1.30	Virtual IP address for HAProxy/keepalived
srv31.hl.test	10.11.1.31	Kubernetes control plane
srv32.hl.test	10.11.1.32	Kubernetes control plane
srv33.hl.test	10.11.1.33	Kubernetes control plane
srv34.hl.test	10.11.1.34	Kubernetes worker node
srv35.hl.test	10.11.1.35	Kubernetes worker node
srv36.hl.test	10.11.1.36	Kubernetes worker node

Homelab Infrastructure

Kubernetes environment runs on three KVM hypervisors. The goal is to maintain service in the event of a loss of a (single) host. This blog post explains how to build a multi-master Kubernetes homelab cluster by hand using KVM, PXE boot and kubeadm.

Hardware

Commodity hardware is used to keep costs to a minimum.

Hostname	CPU Cores	RAM (MB)	Storage
kvm1.hl.test	8	28672	120GB SSD
kvm2.hl.test	8	16384	120GB SSD
kvm3.hl.test	8	16384	120GB SSD
truenas.hl.test	4	8192	240GB SSD, 2x 320GB HDDs in RAID 1 for storage pool
mikrotik.hl.test	1	128	128MB
pi.hl.test	1	512	8GB

Guest Provisioning

Previously, provisioning of KVM guests was done by using a PXE boot server with Kickstart templates.

I have since migrated to Packer to make the VM deployment process faster. PXE boot is still used to provision physical hosts (hypervisors).

Shared Storage

A TrueNAS NFS server is used to create persistent volumes claims using democratic-csi.

Other Services

Homelab provides other services to Kubernetes that aren't covered here:

Homelab Root CA

SSL certificates are signed by the homelab CA.

Create your own Certificate Authority (CA) for homelab environment. Run the following a CentOS 7 server:

$ vim /etc/pki/tls/certs/make-dummy-cert
$ openssl req -newkey rsa:2048 -keyout homelab-ca.key -nodes -x509 -days 3650 -out homelab-ca.crt

Create a Kubernetes Wildcard Cert Signed by the Root CA

$ DOMAIN=wildcard.apps.hl.test
$ openssl genrsa -out "${DOMAIN}".key 2048 && chmod 0600 "${DOMAIN}".key
$ openssl req -new -sha256 -key "${DOMAIN}".key -out "${DOMAIN}".csr
$ openssl x509 -req -in "${DOMAIN}".csr -CA homelab-ca.crt -CAkey homelab-ca.key -CAcreateserial -out "${DOMAIN}".crt -days 1825 -sha256

Average Power Consumption

~80W

Monthly, the homelab costs (((80W * 24h) / 1000) * £0.19/kWh * 365days) / 12months = £11.10 (~15$).

Deployment

The deployment section assumes that the homelab environment has been provisioned.

Ansible-defined Kubernetes Homelab

See ansible/README.md.

Use this to deploy Kubernetes cluster with Ansible.

Manage Kubernetes Homelab with Terraform

See terraform/README.md.

Use this to deploy various Kubernetes resources with Terraform.

Manage Kubernetes Homelab Manually

Create a monitoring namespace:

$ kubectl apply -f ./monitoring-ns-istio-injection-enabled.yml

kube-state-metrics

Deploy kube-state-metrics:

$ kubectl apply -f ./kube-state-metrics

Prometheus

Create a secret called prometheus-cluster-name that contains the cluster name the Prometheus instance is running in:

$ kubectl -n monitoring create secret generic \
  prometheus-cluster-name --from-literal=CLUSTER_NAME=kubernetes-homelab

Deploy prometheus:

$ kubectl apply -f ./prometheus

Grafana

Deploy grafana:

$ kubectl apply -f ./grafana

Alertmanager

Alertmanager uses the Incoming Webhooks feature of Slack, therefore you need to set it up if you want to receive Slack alerts.

Update the config map alertmanager/alertmanager-config-map.yml and specify your incoming webhook URL. Deploy alertmanager:

$ kubectl apply -f ./alertmanager

Mikrotik-exporter

Update the secret file mikrotik-exporter/mikrotik-exporter-secret.yml and specify your password for the Mikrotik API user. Deploy mikrotik-exporter:

$ kubectl apply -f ./mikrotik-exporter

X509 Certificate Exporter

Deploy the Helm chart:

$ helm install x509-certificate-exporter \
  enix/x509-certificate-exporter \
  --namespace monitoring \
  --values ./x509-certificate-exporter/values.yml

MetalLB

Update the config map metallb/metallb-config-map.yml and specify the IP address range. Deploy MetalLB network load-balancer:

$ kubectl apply -f ./metallb

Install Istio

The Istio namespace must be created manually.

$ kubectl create ns istio-system

The kubectl apply command may show transient errors due to resources not being available in the cluster in the correct order. If that happens, simply run the command again.

kubectl apply -f ./istio/istio-kubernetes.yml

Install httpd-healthcheck:

$ kubectl apply -f ./httpd-healthcheck

Install Istio Addons - Prometheus

$ kubectl apply -f istio-addons/prometheus

Install Istio Addons - Kiali

$ kubectl apply -f istio-addons/kiali

lbrines / kubernetes-homelab Goto Github PK