GithubHelp home page GithubHelp logo

terraform-hcloud-k3s's Introduction

Kubernetes (k3s) Terraform installer for Hetzner Cloud

This Terraform module creates a Kubernetes Cluster on Hetzner Cloud infrastructure running Ubuntu 20.04. The cluster is designed to take advantage of the Hetzner private network, and is equiped with Hetzner specific cluster enhancements.

Cluster size and instance types are configurable through Terraform variables.

Install

Prerequisites

  • Terraform must be installed
  • Bash must be installed
  • SSH should be installed and configured with an SSH Key and Agent (Recommended)
  • Having kubectl installed is recommended

Note that you'll need Terraform v0.12 or newer to run this project.

Hetzner Cloud API Token

Before running the project you'll have to create an access token for Terraform to connect to the Hetzner Cloud API.

read -sp "Hetzner Cloud API Token: " TF_VAR_hcloud_token # Enter your Hetzner Cloud API Token (it will be hidden)
export TF_VAR_hcloud_token

Usage

Create a main.tf file in a new directory with the following contents:

variable "hcloud_token" {
  type = string
}

provider "hcloud" {
  token = var.hcloud_token
}

# Create a new SSH key
resource "hcloud_ssh_key" "default" {
  name = "Terraform Example"
  public_key = file("~/.ssh/id_rsa.pub")
}

module "cluster" {
  source  = "cicdteam/k3s/hcloud"
  version = "0.1.2"
  hcloud_token = var.hcloud_token
  ssh_keys = [hcloud_ssh_key.default.id]
}

output "master_ipv4" {
  depends_on  = [module.cluster]
  description = "Public IP Address of the master node"
  value       = module.cluster.master_ipv4
}

output "nodes_ipv4" {
  depends_on  = [module.cluster]
  description = "Public IP Address of the worker nodes"
  value       = module.cluster.nodes_ipv4
}

That's all it takes to get started!

Pin to a specific module version using version = "..." to avoid upgrading to a version with breaking changes. Upgrades to this module could potentially replace all master and worker nodes resulting in data loss. The terraform plan will report this, but it may not be obvious.

Create an Hetzner Cloud Kubernetes cluster with one master and a node:

terraform apply

This will do the following:

  • provisions Hetzner Cloud Instances with Ubuntu 20.04 (the instance type/size of the master and the node may be different)
  • installs K3S components and supporting binaries
  • joins the nodes in the cluster
    • installs Hetzner Cloud add-ons:
      • CSI (Container Storage Interface driver for Hetzner Cloud Volumes)
      • CCM (Kubernetes cloud-controller-manager for Hetzner Cloud)
  • creates two bash scripts to setup/destroy new context in the kubectl admin config file for local kubectl

After applying the Terraform plan you'll see several output variables like the master public IP and nodes IPs.

terraform destroy -force

Be sure to clean-up any CSI created Block Storage Volumes, and CCM created NodeBalancers that you no longer require.

Addons Included

The Hetzner Cloud cloud controller manager integrates your Kubernets cluster with the Hetzner Cloud API. Read more about kubernetes cloud controller managers in the kubernetes documentation.

Features

  • instances interface: adds the server type to the beta.kubernetes.io/instance-type label, sets the external ipv4 and ipv6 addresses and deletes nodes from Kubernetes that were deleted from the Hetzner Cloud.
  • zones interface: makes Kubernetes aware of the failure domain of the server by setting the failure-domain.beta.kubernetes.io/region and failure-domain.beta.kubernetes.io/zone labels on the node.
  • Private Networks: allows to use Hetzner Cloud Private Networks for your pods traffic.
  • Load Balancers: allows to use Hetzner Cloud Load Balancers with Kubernetes Services

This is a Container Storage Interface driver for Hetzner Cloud enabling you to use Volumes within Kubernetes.

When a PV is deleted, the Hetzner Block Storage Volume will be deleted as well, based on the ReclaimPolicy.

Learn More about Persistent Volumes on kubernetes.io.

terraform-hcloud-k3s's People

Contributors

cicdteam avatar mikebell avatar tabsvsspaces avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

terraform-hcloud-k3s's Issues

Error syncing load balancer

Hi, I'm pretty new to Terraform and k8s so forgive me if I'm missing something obvious, but when trying to install Helm charts I keep running into the same error:

Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.Create: neither load-balancer.hetzner.cloud/location nor load-balancer.hetzner.cloud/network-zone set

Create full kubeconfig file at location

It would be nice to have an option where it will just create a local standalone kubeconfig file without modifying global system configuration.

My usual tooling is that I then set the KUBECONFIG environment variable when inside the cluster folder using https://direnv.net.

Feature request: disable PW auth over SSH

Hi,

how about using the template scripts to also disable SSH password auth with something like

sed -i "s/#PasswordAuthentication yes/PasswordAuthentication no/" /etc/ssh/sshd_config

Setup fails on a clean install

Hi,

I'm testing this provider and it fails during terraform init for me:

  • clean setup, new hetzner cloud project, exported token
  • copy pasted the main.tf from the README without changes

When running init I get:

$ terraform init
Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hetznercloud/hcloud...
- Finding latest version of hashicorp/hcloud...
- Finding latest version of hashicorp/local...
- Finding latest version of hashicorp/random...
- Finding latest version of hashicorp/template...
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hetznercloud/hcloud v1.27.2...
- Installed hetznercloud/hcloud v1.27.2 (signed by a HashiCorp partner, key ID 5219EACB3A77198B)
- Installing hashicorp/local v2.1.0...
- Installed hashicorp/local v2.1.0 (signed by HashiCorp)
- Installing hashicorp/random v3.1.0...
- Installed hashicorp/random v3.1.0 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/hcloud: provider registry registry.terraform.io does not have a provider named
│ registry.terraform.io/hashicorp/hcloud
│ 
│ Did you intend to use hetznercloud/hcloud? If so, you must specify that source address in each module which requires that provider. To see which modules are currently
│ depending on hashicorp/hcloud, run the following command:
│     terraform providers

Anything I'm missing ?
thanks!

EDIT: I also ran terraform providers as instructed and I get this:

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/hcloud]
└── module.cluster
    ├── provider[registry.terraform.io/hashicorp/template]
    ├── provider[registry.terraform.io/hetznercloud/hcloud]
    ├── provider[registry.terraform.io/hashicorp/local]
    ├── provider[registry.terraform.io/hashicorp/random]
    ├── module.kubeconfig
    │   ├── provider[registry.terraform.io/hashicorp/local]
    │   └── provider[registry.terraform.io/hashicorp/template]
    ├── module.master
    │   ├── provider[registry.terraform.io/hetznercloud/hcloud]
    │   └── provider[registry.terraform.io/hashicorp/template]
    └── module.node_group
        ├── provider[registry.terraform.io/hetznercloud/hcloud]
        └── provider[registry.terraform.io/hashicorp/template]

LVM partitions

I would like to partition the nodes with LVM, so that I can use TopoLVM and Rook-Ceph with Local Storage.

In fact, I don't mind how the base Operating System is installed, but the remaining free space I would like to be managed by LVM.

What would be the correct approach for this?

cluster_name is case-sensitive

If the cluster_name contains upper case letters, routes can not be created.

Steps to reproduce:

  1. create minimal cluster (follow "usage" part of README.md)
...
module "cluster" {
  source  = "cicdteam/k3s/hcloud"
  version = "0.1.2"
  hcloud_token = var.hcloud_token
  cluster_name = "helloWORLD"
}
...
  1. Use cluster_name with lower and uppercase letters e.g. "helloWORLD"
  2. terraform apply

connect to the master and get all nodes using kubectl get nodes.
describe one node using kubectl describe node <nodename> e.g. kubectl describe node helloworld-cx21-0.

the following error / warning appears:

Events:
  Type     Reason                   Age                    From              Message
  ----     ------                   ----                   ----              -------
  Normal   Starting                 5m26s                  kubelet           Starting kubelet.
  Warning  InvalidDiskCapacity      5m26s                  kubelet           invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  5m26s (x2 over 5m26s)  kubelet           Node helloworld-master status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    5m26s (x2 over 5m26s)  kubelet           Node helloworld-master status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     5m26s (x2 over 5m26s)  kubelet           Node helloworld-master status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  5m26s                  kubelet           Updated Node Allocatable limit across pods
  Normal   Starting                 5m24s                  kube-proxy        Starting kube-proxy.
  Normal   NodeReady                5m16s                  kubelet           Node helloworld-master status is now: NodeReady
  Warning  FailedToCreateRoute      4m58s                  route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 434.609599ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      4m48s                  route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 430.176207ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      4m38s                  route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 573.039188ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      4m27s                  route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 1.146160095s: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      4m14s                  route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 2.801960525s: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      4m6s                   route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 501.629225ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      3m55s                  route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 1.588634126s: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      3m47s                  route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 607.737527ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      3m38s                  route_controller  Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 240.739195ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found
  Warning  FailedToCreateRoute      57s (x16 over 3m27s)   route_controller  (combined from similar events): Could not create route 3b2437df-b408-4059-bc3b-5cd304b4abba 10.42.0.0/24 for node helloworld-master after 1.540429998s: hcloud/CreateRoute: hcops/AllServersCache.ByName: hcops/AllServersCache.getCache: not found

Could not retrieve the list of available versions for provider hashicorp/hcloud: provider registry registry.terraform.io does not have a provider named │ registry.terraform.io/hashicorp/hcloud

On Terraform v1.0.10 during command terraform init, I have an error:

│ Error: Failed to query available provider packages

│ Could not retrieve the list of available versions for provider hashicorp/hcloud: provider registry registry.terraform.io does not have a provider named
│ registry.terraform.io/hashicorp/hcloud

│ Did you intend to use hetznercloud/hcloud? If so, you must specify that source address in each module which requires that provider. To see which modules are currently
│ depending on hashicorp/hcloud, run the following command:
│ terraform providers

As a fix I had to modify main.tf file and add following

terraform {
  required_providers {
    hcloud = {
      source = "hetznercloud/hcloud"
      version = "1.32.0"
    }
  }
}

Install specific CSI/CCM manifest via terraform

This would allow to define/update the version of the manifests without changing this module.

I once did this for CSI somewhere with the kubectl provider:

data "http" "csi_driver_manifest" {
  url = "https://raw.githubusercontent.com/hetznercloud/csi-driver/${var.csi_driver_version}/deploy/kubernetes/hcloud-csi.yml"
}

data "kubectl_file_documents" "csi_driver" {
  content = data.http.csi_driver_manifest.body
}

resource "kubectl_manifest" "csi_driver" {
  count      = length(data.kubectl_file_documents.csi_driver.documents)
  yaml_body  = element(data.kubectl_file_documents.csi_driver.documents, count.index)
}

For CMM this would then also make sense.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.