stigok / kubernetes-cluster

Standalone bare-metal kubernetes cluster setup notes. *Closed*: K8s is a fast-moving target. Can't guarantee this works anymore. Archiving.

My Kubernetes cluster bootstrap configuration. Following this guide from start to finish should get you a working cluster with all of the mentioned add-ons.

Tested on Ubuntu 18.04 (Bionic Beaver)

Cluster setup

Determine and save your node's local IP address to a variable before continuing.

export NODE_LOCAL_IP=<local ip>

Step by step...

apt update
apt-get install apt-transport-https ca-certificates curl software-properties-common curl

# Docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Kubernetes
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' > /etc/apt/sources.list.d/kubernetes.list

apt-get update
apt-get install -y kubelet kubeadm kubectl docker-ce=18.06.0~ce~3-0~ubuntu
apt-mark hold kubelet kubeadm kubectl docker-ce

# Prepare for CNI
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.d/99-flannel.conf
echo 'net.bridge.bridge-nf-call-iptables = 1' >> /etc/sysctl.d/99-flannel.conf

# Kubernetes
kubeadm config images pull
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=${NODE_LOCAL_IP}

# A must for single node setup
kubectl taint nodes --all node-role.kubernetes.io/master-

# A must for life
kubectl completion bash >> /etc/bash_completion.d/kubernetes

# Docker config
cat <<EOF > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

Flannel (CNI) v0.10.0

kubectl apply -f flannel/

Cluster add-ons

metrics-server v0.3.1

This is required for node and pod stats for kubectl top pod and the dashboard.

Must update metrics-server-deployment.yaml with the master node hostname and local IP before deploying.

Warning: An extra argument --kubelet-insecure-tls is supplied to make this work. The underlying issue should be fixed.

$ kubectl create -f metrics-server/

ingress-nginx v0.20.0

Apply manifests

$ kubectl apply ingress-nginx/

cert-manager v0.5.2

There's a bug in cert-manager that requires you to disable client validation when applying the manifests.

Requires manual intervention before deploying:

• Update the spec.acme.email field of 90-letsencrypt-cluster-issuers.yaml

Creates ClusterIssuer resources for Let's Encrypt (production and staging).

Apply manifests

$ kubectl create -f cert-manager/ --validate=false

MetalLB v0.7.3

Apply manifests

$ kubectl apply -f metallb/

Add Layer2 configuration to specify the pool of addresses your load balancers will pick from.

The name of the pool can be whatever. Addresses is a list of ranges or single addresses or subnets in CIDR notation (e.g. 1.2.3.4/32, 192.168.42.0/24 or 10.1.2.10-10.1.2.20).

Scaleway: Incoming traffic to VM's are sent to the private IP address, not the external one.

Consult https://metallb.universe.tf/configuration/ for more information.

$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - <address or subnet 1>
EOF

Useful commands

Create a new cluster user

Run this on the master. Will output a complete kubectl configuration file.

$ kubeadm alpha phase kubeconfig user --client-name <username>

Set cluster-admin role for user

Run on master to give username cluster-admin privileges.

$ kubectl create clusterrolebinding root-cluster-admin-binding --clusterrole=cluster-admin --user=<username>

References

• https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
• https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
• https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/
• https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet
• kubernetes-sigs/metrics-server#131 (comment)
• https://kubernetes.io/docs/reference/access-authn-authz/rbac/#permissive-rbac-permissions
• https://cert-manager.readthedocs.io/en/latest/tutorials/acme/securing-nginx-ingress-with-letsencrypt.html

kubernetes-cluster's Issues

Avoid using --kubelet-insecure-tls

References

• kubernetes-sigs/metrics-server#131 (comment)

Create cloudinit config

I want to have a complete cloudinit configuration for this setup.
At least for all the packages and gpg keys.