onedr0p / k3s-homeops-ansible

Bootstrap a k3s cluster on top of Ubuntu 20.04

k3s-homeops-ansible

This is an opinionated way to provision Ubuntu 20.04 and install k3s on top.

Prerequisites

There's a couple things that will need to be done before you get starting running with Ansible.

1. Install Ansible >= 2.10.0 on your local machine
2. Install a supported OS on each of your nodes
3. Set static IP for each node on the OS or in your router (you can use the IP assigned via DHCP, but it's not recommended)
4. Copy your local public ssh key with ssh-copy-id to each node
5. Optional review playbooks and roles to understand what these Ansible script will do

After that we're ready to continue with Ansible...

Ansible

Get started by cloning this repository and copying the hosts and config into a new directory.

# clone this repo
git clone https://github.com/onedr0p/k3s-homeops-ansible
# change into the directory
cd k3s-homeops-ansible
# copy the hosts and config to a new folder
cp -r ./inventory/local ./inventory/custom

Update the Ansible config files

Note: This project uses PyratLabs/ansible-role-k3s for installing k3s. Configuration options can be viewed in their README.

After you have copied over the configuration files you will need to update the configuration in the files:

• ./inventory/custom/hosts.yml: Host definitions
• ./inventory/custom/host_vars/*.yml: Host IP and host level variables
• ./inventory/custom/group_vars/*.yml: Global variables for all hosts

Each file it carefully documented.

Get Ansible dependencies

ansible-galaxy install -r requirements.yml

Run the playbooks

# This playbook will prepare your nodes for Kubernetes
ansible-playbook -i ./inventory/custom/hosts.yml ./playbooks/os-build.yml
# This playbook will install k3s
ansible-playbook -i ./inventory/custom/hosts.yml ./playbooks/cluster-build.yml

Verify the cluster is up and running

kubectl --kubeconfig ./kubeconfig get nodes -o wide

k3s-homeops-ansible's Issues

Docker should be optional since k3s could install and use containerd

Document supported CNI

Probably only want to stay with calico and flannel for now.

Implement longhorn role

Create vagrant task to remove old block devices

Running into errors where vagrant thinks the block device exists

VBoxManage: error: Failed to create medium
VBoxManage: error: Could not create the medium storage unit '/Users/devin/Code/k3s-cluster-ansible/k8s-node-10-block.vdi'.
VBoxManage: error: VDI: cannot create image '/Users/devin/Code/k3s-cluster-ansible/k8s-node-10-block.vdi' (VERR_ALREADY_EXISTS)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component MediumWrap, interface IMedium
VBoxManage: error: Context: "RTEXITCODE handleCreateMedium(HandlerArg *)" at line 510 of file VBoxManageDisk.cpp

hashicorp/vagrant#8107 (comment)

Maybe we should add this in a task to run if shit gets fubared?

Install rook-ceph pre-reqs

https://rook.io/docs/rook/v1.2/k8s-pre-reqs.html

tl;dr

sudo apt-get install -y lvm2

and yoink this from my repo

https://github.com/onedr0p/k3s-gitops/blob/master/ansible/roles/system/tasks/main.yml#L6

Add taint to storage nodes for rook-ceph

Would be nice to taint the servers running rook-ceph and apply tolerations to the rook-ceph deployments. This way standard workloads can only be ran on the worker nodes without any affinity rules.

Move download/tasks into k3s role

Implement E2E Testing with GitHub Actions

https://github.com/hashicorp/terraform-github-actions

Explore secret management in GitHub.

Add description of project to README

• What is the goal of this project?
• What components are defaulted?
• What dependencies are needed to get started?

k3s on VMware vSphere

FYI I started to work on the k3s on vSphere. I have the infra deployment kinda done using terraform.
I think would be good to join forces here.

https://github.com/likamrat/k3s_vmware

RFC: Switch to Debian

Explore Prow for PR approvals

https://github.com/kubernetes/test-infra/tree/master/prow

Let timezone be configurable

Right now it is hardcoded in the ansible scripts.

Convert Calico BGP file to inline with EOF

Getting errors because it references a local path that is no longer local.

failed: [k3s-0 -> localhost] (item=kubectl apply -f /home/ryan/src/k3s-bootstrap/devops/kube-system/calico/calico.yaml) => {"ansible_loop_var": "item", "attempts": 6, "changed": false, "cmd": ["kubectl", "apply", "-f", "/home/ryan/src/k3s-bootstrap/devops/kube-system/calico/calico.yaml"], "delta": "0:00:00.239140", "end": "2020-02-17 00:55:58.162550", "item": "kubectl apply -f /home/ryan/src/k3s-bootstrap/devops/kube-system/calico/calico.yaml", "msg": "non-zero return code", "rc": 1, "start": "2020-02-17 00:55:57.923410", "stderr": "Unable to connect to the server: x509: certificate signed by unknown authority", "stderr_lines": ["Unable to connect to the server: x509: certificate signed by unknown authority"], "stdout": "", "stdout_lines": []}

Project: Simplify Project

• Keep a reduced number of options in our config.yaml
• Remove Docker install and stay with containerd, maybe use https://github.com/GoogleContainerTools/kaniko for building images if people want Drone CI or something.

Explore Ansible Lint

https://github.com/ansible/ansible-lint

Also setup in pre-commit: https://github.com/ansible/ansible-lint/blob/master/.pre-commit-hooks.yaml

Allow option for over-writing kube config

toggle with variable, default to no

Add a preflight playbook

Ensure tools are installed locally

• helm
• kubectl
• fluxctl
• calicoctl
• vault

...

Prompt to copy sample files if not exists

• hosts
• main variables
• encrypted vault variables
• unencrypted vault variables

...

• Validate that a key file is created or offer to generate for Ansible Vault

...

• Warn that the option to encrypt the encrypted variables should be set to true as part as playbook

...

• Validate that pre-commit includes option to check for Ansible-Vault being encrypted
• Validate that .gitattributes includes filter for git-crypt and the vault.key file

Output Vault root token & unseal token to repo root

Needed to update vault in other stages.

Add to gitignore too.

Add kubeconfig to .gitignore

Document global Ansible variables and their values

add-repo-key script should have username configurable

Prep Disks for Ceph

Ceph isn't always picking up block devices at initial provision. Need to add detection for state of block device with option to clean.

Add ansible tasks to taskfile

Local Path Provisioner for k3s should be configurable

Add Kubeval to CI lint

https://github.com/instrumenta/kubeval

Implement linting workflows in CI

Add install homebrew deps to taskfile

Something like ... task macos:deps

Do not send poweroff on power button press

edit /etc/systemd/logind.conf

change #HandlePowerKey=poweroff to HandlePowerKey=ignore

this will prevent misclicks with the power button on my NUCs, you can still force poweroff by holding the button

Document supported CRI

Docker and containerd

Implement pre-commit

Explore adding local user variable

Default to {{ ansible_user }} and update any local copy jobs to use that user instead.

Use case is for build processes where the remote user is different from the local user.

Swap out hardcoded server names with variables in taints

Currently set to r610... etc

Remove more packages

apt-get --purge -qqy remove apport bcache-tools btrfs-progs byobu cloud-guest-utils cloud-initramfs-copymods cloud-initramfs-dyn-netconf friendly-recovery fwupd landscape-common lxd-agent-loader ntfs-3g open-vm-tools plymouth plymouth-theme-ubuntu-text popularity-contest snapd sosreport tmux ubuntu-advantage-tools ufw```

Pass k3s flags as a Ansible variable when installing k3s master or nodes

Might make configuring more dynamic to allow a variable to pass any k3s install flags to

https://rancher.com/docs/k3s/latest/en/installation/install-options/

Minor consideration would be to check if docker flag is included and then install docker.

This would also cover issues #6 and #5

Create all.yml.sample and gitignore current all.yml

Makes it so having settings specific to our envs for testing, but requires us to update the sample when new vars are added or removed.

Vault Values will not work

They assume local access to the .txt files.

Explore “Kuby” for environment variable substitution

Ref: https://blog.smartive.ch/how-we-simplified-our-kubernetes-deployments-with-an-alternative-to-helm-aafedcfd4cce

Implement linting

• ansible
• yaml
• shellcheck

Calico currently enabled, flannel should be configurable to use instead

Related to BGP mode.

Vault Configuration should wait for Consul

I believe it's initializing before consul is ready

Ansible with Vagrant currently failing on this step

TASK [k3s/master : Restore node-token file access] *****************************
changed: [192.168.88.200]
ERROR! Attempting to decrypt but no vault secrets found
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.