scholzj / terraform-aws-kubernetes

Terraform module for Kubernetes setup on AWS

AWS Kubernetes

AWS Kubernetes is a Kubernetes cluster deployed using Kubeadm tool. It provides full integration with AWS. It is able to handle ELB load balancers, EBS disks, Route53 domains etc.

• AWS Kubernetes
  • Updates
  • Prerequisites and dependencies
  • Including the module
  • Add-ons
  • Custom add-ons
  • Tagging

Updates

• 1.4.2023 Update to Kubernetes 1.26.3 + update add-ons (Ingress-NGINX Controller, External DNS, Metrics Server, AWS EBS CSI Driver)
• 4.3.2023 Update to Kubernetes 1.26.2 + update add-ons (Ingress-NGINX Controller)
• 22.1.2023 Update to Kubernetes 1.26.1 + update add-ons (External DNS)
• 10.12.2022 Update to Kubernetes 1.26.0 + update add-ons (AWS EBS CSI Driver, Metrics server)
• 13.11.2022 Update to Kubernetes 1.25.4 + update add-ons
• 2.10.2022 Update to Kubernetes 1.25.2 + update add-ons
• 26.8.2022 Update to Kubernetes 1.25.0 + Calico upgrade
• 22.8.2022 Update to Kubernetes 1.24.4
• 16.7.2022 Update to Kubernetes 1.24.3
• 27.6.2022 Update to Kubernetes 1.24.2
• 11.6.2022 Update to Kubernetes 1.24.1 + update add-ons + remove dependency on the template provider
• 8.5.2022 Update to Kubernetes 1.24.0 + update add-ons
• 23.3.2022 Update to Kubernetes 1.23.5 + update add-ons
• 19.2.2022 Update to Kubernetes 1.23.4
• 12.2.2022 Update to Kubernetes 1.23.2
• 29.12.2021 Update to Kubernetes 1.23.1
• 11.12.2021 Update to Kubernetes 1.23.0

Prerequisites and dependencies

• AWS Kubernetes deploys into existing VPC / public subnet. If you don't have your VPC / subnet yet, you can use this module to create one.
  • The VPC / subnet should be properly linked with Internet Gateway (IGW) and should have DNS and DHCP enabled.
  • Hosted DNS zone configured in Route53 (in case the zone is private you have to use IP address to copy kubeconfig and access the cluster).
• To deploy AWS Kubernetes there are no other dependencies apart from Terraform. Kubeadm is used only on the EC2 hosts and doesn't have to be installed locally.

Including the module

Although it can be run on its own, the main value is that it can be included into another Terraform configuration.

module "kubernetes" {
  source = "scholzj/kubernetes/aws"

  aws_region    = "eu-central-1"
  cluster_name  = "aws-kubernetes"
  master_instance_type = "t2.medium"
  worker_instance_type = "t2.medium"
  ssh_public_key = "~/.ssh/id_rsa.pub"
  ssh_access_cidr = ["0.0.0.0/0"]
  api_access_cidr = ["0.0.0.0/0"]
  min_worker_count = 3
  max_worker_count = 6
  hosted_zone = "my-domain.com"
  hosted_zone_private = false

  master_subnet_id = "subnet-8a3517f8"
  worker_subnet_ids = [		
      "subnet-8a3517f8",
      "subnet-9b7853f7",
      "subnet-8g9sdfv8"
  ]
  
  # Tags
  tags = {
    Application = "AWS-Kubernetes"
  }

  # Tags in a different format for Auto Scaling Group
  tags2 = [
    {
      key                 = "Application"
      value               = "AWS-Kubernetes"
      propagate_at_launch = true
    }
  ]
  
  addons = [
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/storage-class.yaml",
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/heapster.yaml",
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/dashboard.yaml",
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/external-dns.yaml",
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/autoscaler.yaml"
  ]
}

An example of how to include this can be found in the examples dir.

Add-ons

Currently, following add-ons are supported:

• Kubernetes dashboard
• Heapster for resource monitoring
• Storage class and CSI driver for automatic provisioning of persistent volumes
• External DNS (Replaces Route53 mapper)
• Ingress
• Autoscaler

The add-ons will be installed automatically based on the Terraform variables.

Custom add-ons

Custom add-ons can be added if needed. For every URL in the addons list, the initialization scripts will automatically call kubectl -f apply <Addon URL> to deploy it. The cluster is using RBAC. So the custom add-ons have to be RBAC ready.

Tagging

If you need to tag resources created by your Kubernetes cluster (EBS volumes, ELB load balancers etc.) check this AWS Lambda function which can do the tagging.

terraform-aws-kubernetes's Issues

private subnet

There is some easy way to deploy the cluster in a private subnet ?

Nodes regularly failing after a day

Hi there,

I understand this is not a bug with your implementation.

We are seeing an issue with nodes failing to start pods with the following errors.

MountVolume.SetUp failed for volume "default-token-hrpk9" : mount failed: exit status 1 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f570149c-5462-11e8-a72b-02a9a02d1274/volumes/kubernetes.io~secret/default-token-hrpk9 --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/f570149c-5462-11e8-a72b-02a9a02d1274/volumes/kubernetes.io~secret/default-token-hrpk9 Output: Failed to start transient scope unit: Connection timed out

SystemD has become unresponsive.

We have a few CronJobs running which I suspect is causing the systemd to eventually be unable to mount new secrets. Increasing the CronJob rate hasn't made much affect.

The closest issue I have seen is kubernetes/kubernetes#57345 but the issue varies slightly. Remoting into the machine systemd is completely unresponsive.

systemctl list-units --all | wc -l
Failed to list units: Connection timed out
0

What I know

1. System reboot fixes the issue.
2. Systemd is an older version, and unlikely to be updated anytime soon
3. Possibly not a kubernetes issue

Any suggestions on where to hunt?

Why centos?

Hi!
Thank you for your effort!

But why centos?

There is lock to centos marketplace ami, so no flexibility +
ASG Error: Launching a new EC2 instance. Status Reason: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=aw0evgkw8e5c1q413zgy5pjce. Launching EC2 instance failed.

From AWS: Recommendation to not use launch configurations

"will stop supporting new EC2 instances types after December 31, 2022" I wonder what that means?

Core DNS addon

Hi @scholzj,

Core DNS has reached General Availability in Kubernetes 1.11: https://kubernetes.io/blog/2018/07/10/coredns-ga-for-kubernetes-cluster-dns/

Do you have plan to support Core DNS as an addon in the future ?

Nicolas

Install script didn't run?

Hi! I used your module but after terrfaorm apply I have clear master instance on EC2. Even no epel-release have installed.
What it could be?

Tags must be a list instead of map

Terraform supports defining tags as a map instead of a list of maps (like shown in the example usage).

This means situations like this are valid:

module "kubernetes" {
  tags = "${merge(
    map("Name", format("%v-kubernetes", var.vpc_name)),
    map("Country", substr(var.region, 0, 2)),
    map("DataCenter", substr(var.region, 3, length(var.region) - 5)),
    var.tags
  )}"

  tags2 = {
    Name = "${var.vpc_name}-kubernetes"
    Country = "ca"
    DataCenter = "central"
  }
}

However, trying these situations does not work and complains returning:

* module.management.module.kubernetes.var.tags2: variable tags2 in module management.kubernetes should be type list, got map

Scaling down fails due to missing permission

It seems that Kubernetes is failing to scale down an autoscaling group. The permission I could see that it seems to be missing is autoscaling:DescribeLaunchConfigurations. Not sure if it needs more than just that one.

Can I chose the kubernetes version ?

I like to choose to install kubernetes 1.15....there is some easy for this ?

Error: Attribute redefined - The argument "tags" was already set

I'm receiving the following error on module version 1.6.1:

Error: Attribute redefined

  on .terraform/modules/kubernetes/scholzj-terraform-aws-kubernetes-6fb01f4/main.tf line 371, in resource "aws_autoscaling_group" "nodes":
 371:   tags = [{

The argument "tags" was already set at
.terraform/modules/kubernetes/scholzj-terraform-aws-kubernetes-6fb01f4/main.tf:365,3-7.
Each argument may be set only once.


Error: Attribute redefined

  on .terraform/modules/kubernetes/scholzj-terraform-aws-kubernetes-6fb01f4/main.tf line 377, in resource "aws_autoscaling_group" "nodes":
 377:   tags = ["${var.tags2}"]

The argument "tags" was already set at
.terraform/modules/kubernetes/scholzj-terraform-aws-kubernetes-6fb01f4/main.tf:365,3-7.
Each argument may be set only once.

It seems the tags attribute shouldn't be redeclared over and over here:

terraform-aws-kubernetes/main.tf

Lines 356 to 382 in 7e4342c

	resource "aws_autoscaling_group" "nodes" {
	vpc_zone_identifier = ["${var.worker_subnet_ids}"]
	name = "${var.cluster_name}-nodes"
	max_size = "${var.max_worker_count}"
	min_size = "${var.min_worker_count}"
	desired_capacity = "${var.min_worker_count}"
	launch_configuration = "${aws_launch_configuration.nodes.name}"
	tags = [{
	key = "Name"
	value = "${var.cluster_name}-node"
	propagate_at_launch = true
	}]
	tags = [{
	key = "kubernetes.io/cluster/${var.cluster_name}"
	value = "owned"
	propagate_at_launch = true
	}]
	tags = ["${var.tags2}"]
	lifecycle {
	ignore_changes = ["desired_capacity"]
	}
	}

Fix kubelet configuration section in init-aws-kubernetes-node.sh does thing

/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf from my server:

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS              

The sed lines in the Fix kubelet configuration section wont match with anything

Nodes are flapping between ready and not ready state

I am trying to launch kubernetes cluster using your code. But I am facing flapping issue same as kubernetes/kubernetes#45419 . Also when nodes are in not ready state, i got cpu spikes.

Launch configuration is invallid

Failed to launch a new EC2 instance. User data for the launch configuration is also very strange:

���ÿ¤VmoÛ6�þ. ÿáæ�H;�bÝfi�Ö�ÒÆé�&ñ�'�¶a3hê$³¦H�<¹VÑ�?P/��d@°ù�Áós÷ÜñÞôÎ�BCìªvx�e¥I9á��j�ÙkXØÊdÂ×ãÁù�óÉÛéõÅÉñåï�$ÞØ'ôAYs�£ôé^²�0¶�ÚKzÛ'*8��5XA$ä²DC¯!W��(q<PF��_�[U�ô�	�36Ã4,�7�®¼0!GÏ&FÚL�â�^,�í�(�7Ä7,,Që ½r´��«�ï8üè�¾P�/DX&I@�fa�~a�öWô�7�ú«S�s¡t�àÆYOðñúíäøä|~5ý8¹�»zñR�¤í��¾Ò«ÃÏ«çv�>¹�Í/�Ï'ãïCM¥®�¡OKëQ8�J[î�\^L®&³ù§ÉåìÃôb<�¥¯Òg�$y�3$ ¥
`�®Aä����)S-�2ÌE¥)ôþ�[���U�C �¥ ¹l K�(&�pãP�f°¨!ú(²��ù�S��ÿ5Rrz}v6ÿe:»j¢��ËÊk`��Dî�óÑá«ôÙO�iwr-��ñ�I°L�à½å'MXçb�ñ±@Û/è¥��l�nø�åÒÂ`ØK�ð
ÈÃþ1ûc�ö�ûºÿ$Úø`�	­!³r�>©«�T'b5ÔUÉ*R:@�k%��Â9ôÌÅ�	M1E¿@¯ËgQ�IkrU°R�Q �ÆD�1�Î6!�#Î3ûÅh+²´e�IäZ�jÃ%�²�·r&1�z�G¥X¡�r���@·�Ü*ìÆóq[6 mé¬A�ó[e�¤ xóf2=���#I^WeÃ�Ò�ï�[Cÿç�௤éÂ�ËÉB�¬¼�÷±9!W¢h*ÕVYZX[hl"¬«�7�;�õ�¶yy8?<HÐ��Æl<J
WÈ%ÊÕx�D�ùνpÅ
ë�²e¶	�­°N�W$Ðý�®í]É:ÔÖÊdz���Mn½Dxz;�]¥³áÝ�ì»ä_ÿë^E���yª6½5h«ªò"�Ç$6�S°�8c²ð¶r,ój�~�ê@Xfwäí-�¼Øo3Þ�Û�w<i@�«<Íøè)ëÜM#ù��OÌZykâd��b g�«ù»÷�Óë_çÇ�ïg\Áý�³éõI��3Ö<9sÞ®U�>�¹Áÿu,ðá}ÞïòýÀÿü2q¼�ð��$$­ª$
mU÷³åF���;â�ÞOËÛø^Þ
ôn��0�Y;gO¶�£�AJ�^e�v�39�Bk¦�E¦0�=�{¸�we)Ñ�Ê��Ó¹ÙÎ`
¼CCÓY¢òf¾4où£G�" |�£�¿�ÉÜßZ�x�W�iÚÓ�À|Þå!h�y���í�¦Ò�oàÒÝ�]T&�}ìé�v�\%I¿¼<Æ�×ß>[e�1²+4mÅl÷7°æ£�5+nøÝN�íº9:<8x��e*H»F_·¦Xe�È���rL
¶Fß:��ûö·�c{É?�ÿÿªN­©z	

Using your module as follows:

data "aws_availability_zones" "available" {}

resource "aws_vpc" "default" {
  cidr_block = "${var.cidr}"
  instance_tenancy = "dedicated"

  tags {
    Name = "vpc-${var.name}"
  }
}

resource "aws_internet_gateway" "default" {
  vpc_id = "${aws_vpc.default.id}"
}

resource "aws_route" "internet_access" {
  route_table_id = "${aws_vpc.default.main_route_table_id}"
  destination_cidr_block = "0.0.0.0/0"
  gateway_id = "${aws_internet_gateway.default.id}"
}

resource "aws_subnet" "default" {
  count = 2
  vpc_id = "${aws_vpc.default.id}"
  cidr_block = "10.0.${count.index}.0/24"
  map_public_ip_on_launch = true
  availability_zone = "${data.aws_availability_zones.available.names[count.index]}"
}

resource "aws_route_table_association" "default" {
  count = 2
  route_table_id = "${aws_vpc.default.default_route_table_id}"
  subnet_id = "${aws_subnet.default.*.id[count.index]}"
}



module "kubernetes" {
  source = "scholzj/kubernetes/aws"

  aws_region    = "eu-central-1"
  cluster_name  = "aws-kubernetes"
  master_instance_type = "t2.medium"
  worker_instance_type = "t2.medium"
  ssh_public_key = "~/.ssh/id_rsa.pub"
  ssh_access_cidr = ["0.0.0.0/0"]
  api_access_cidr = ["0.0.0.0/0"]
  min_worker_count = 1
  max_worker_count = 2
  hosted_zone = "cluster.moreapp.com"
  hosted_zone_private = false

  master_subnet_id = "${aws_subnet.default.*.id[0]}"
  worker_subnet_ids = "${aws_subnet.default.*.id}"

  # Tags
  tags = {
    Application = "AWS-Kubernetes"
  }

  # Tags in a different format for Auto Scaling Group
  tags2 = [
    {
      key                 = "Application"
      value               = "AWS-Kubernetes"
      propagate_at_launch = true
    }
  ]

  addons = [
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/storage-class.yaml",
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/heapster.yaml",
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/dashboard.yaml",
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/external-dns.yaml",
    "https://raw.githubusercontent.com/scholzj/terraform-aws-kubernetes/master/addons/autoscaler.yaml"
  ]
}

Support for private subnets

Can we create a node in private subnet also, would like to force isolation of some services in a private subnet.

changing the worker_instance_type causes error on terraform apply

Thanks for the great boilerplate! Awesome.

When I changed the worker_instance_type I got the following error:
aws_launch_configuration.nodes: Error creating launch configuration: AlreadyExists

According the docs name_prefix should be used instead of name for the aws_launch_configuration.

Module does not work in multi-account environment

Use the basics of providers limits the ability to setup multi-account environments.
In my case, we split the environment into management, production and development as completely independent accounts, drastically simplifying the overload required to provide access to different sectors of the company, such as contractors.

To address this issue, you can expand the provider configuration to something like this:

provider "aws" {
  alias = "kubernetes"
  region = "${var.region}"
  assume_role {
    session_name = "Terraform"
    role_arn = "arn:aws:iam::${var.account}:role/Admin"
  }
}

And then for every AWS resource, data, etc, you need to define the provider, like this:

resource "aws_iam_policy" "master_policy" {
  provider    = "aws.kubernetes"
  name        = "${var.cluster_name}-master"
  path        = "/"
  description = "Policy for role ${var.cluster_name}-master"
  policy      = "${data.template_file.master_policy_json.rendered}"
}

etcd down after reboot, selinux issue

Hi, thank you for nice terraform k8s setup module,
I've faced the issue - after AWS host reboot, etcd was down, etcd container logs:

2019-05-03 17:48:34.873535 C | etcdmain: cannot access data directory: open /var/lib/etcd/.touch: permission denied

Looks like the config file, which is changed during setup

https://github.com/scholzj/terraform-aws-kubernetes/blob/master/scripts/init-aws-kubernetes-master.sh#L60

is actually a symlink

# ls -la /etc/sysconfig/selinux /etc/selinux/config
-rw-r--r--. 1 root root 543 May  3 18:02 /etc/selinux/config
lrwxrwxrwx. 1 root root  17 May  3 18:02 /etc/sysconfig/selinux -> ../selinux/config

but after the setup script, we have two files

# sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux
# ls -la /etc/sysconfig/selinux /etc/selinux/config
-rw-r--r--. 1 root root 543 May  3 18:02 /etc/selinux/config
-rw-r--r--. 1 root root 544 May  3 18:02 /etc/sysconfig/selinux

This leads to SELinux being enabled again after reboot/upgrade, and k8s stops functioning because of etcd container is down

Module does not support configuring Kubernetes multi-master

This might be stretching a bit your motivations on your module, but so far this is the easiest module to read (and likely easiest to maintain/improve) from my 2 weeks research for setting up your own Kubernetes cluster.

I went through kops, kube-aws, typhoon, juju, etc. All of them are too biased on how your environment needs to be to setup Kubernetes. Then don't have to be like that. Just as one example, there's no way to setup Kubernetes on a pre-existing AWS subnet.

On this module, things are way different. It does what's supposed to do: setup Kubernetes.

Based on that, it'd be amazing to see multi-master support on this module. That requires more setup, such as independently building an etcd cluster (with ASG), potentially being based of a shared EBS volume.

I'll try to describe more (or potentially expand on having a PR). Besides being in a rush to delivery the new infrastructure in 2-3 weeks, a single master would be ok for a couple months for me, but the company I work for would require a multi-master fairly soon.

`vpc_zone_identifier`: should be a list

When attempting to use the output of a counted resource into the module, it throws the error that vpc_zone_identifier should be a list. Example:

module "kubernetes" {
  source = "scholzj/kubernetes/aws"

  master_subnet_id = "${element(aws_subnet.public.*.id, 0)}"
  worker_subnet_ids = "${aws_subnet.protected.*.id}"

  # ...
}

SSH key pair creation should be written to pre-req section of doc

LoadBalancer Service stuck in pending

Nice module.

I tried creating a LoadBalancer service to expose the cluster through and ALB loadbalancer, but it stays in pending state.
Do you have any recommendations for exposing services publicly using this module?

It says Ingress is supported (appears to be nginx-ingress controller) - However I would like to keep my nodes private and use an external loadbalancer to route traffic.

I would love to hear your thoughts on how to expose services publicly. The docs are a bit sparse.

Thank you,
Liam

The connection to the server localhost:8080 was refused

I'm not sure where the .kube file is saved, i'm able to ssh to my master node but when I run kubectl get nodes i this this messege:

'' The connection to the server localhost:8080 was refused - did you specify the right host or port? ''

so I tried opening the port 8080 but still had the same problem. Am I missing anything?