Terraform module to create Amazon Elastic Kubernetes (EKS) resources 🇺🇦
Home Page: https://registry.terraform.io/modules/terraform-aws-modules/eks/aws
License: Apache License 2.0
It is great that with this module I can use more than one ASG worker pool. Would be nice to be able to use spot instances i.e. for background jobs or any applications that can recover very fast from replaced node.
Cannot use spot instances as worker nodes (or at least do not know how)
I could define that one (or all) of my worker nodes ASG are using spot instances.
Should I be exporting my bastion's Ec2 private DNS NAME before I do terraform apply ?
I used bastion host to provision the EKS cluster
I am not clear on EC2PrivateDNSName variable
to change the instance type
* module.eks.aws_launch_configuration.workers: 1 error(s) occurred:
* aws_launch_configuration.workers: Error creating launch configuration: AlreadyExists: Launch Configuration by this name already exists - A launch configuration already exists with the name eks-path-prod-0
status code: 400, request id: xxx-xxx-xxx-xxx-xxx
deploy EKS cluster, then change the instance type and apply again.
Should be no issue.
I am creating an eks cluster with providing a cluster_security_group_id and worker_security_group_id.
When specifying a security group aka cluster_security_group_id = "sg-123" or worker_security_group_id = "sg-123". I get
Error: Error running plan: 1 error(s) occurred:
module.eks.local.cluster_security_group_id: local.cluster_security_group_id: Resource 'aws_security_group.cluster' not found for variable 'aws_security_group.cluster.id'`<br>
Create an eks cluster with a cluster_security_group_id or worker_security_group_id specified.
Terraform does not support short circut evaluation in it's ternary operator. The fix to the issue is specified here
We should be able to create an EKS cluster while specifying cluster sg or worker sg as the documentation currently specifies.
asg size changes should be ignored.
Updated asg sized after deploy, terraform apply detects the changes, which should be ignored.
At least change indesired_capacity should be ignored.
~ module.eks.aws_autoscaling_group.workers
desired_capacity: "2" => "1"
max_size: "5" => "3"
min_size: "2" => "1"
Ignore the changes, since we don't want the running system to be re-sized.
If you are fine to ignore change in desired_capacity, I can raise PR for this feature, please confirm.
kube-proxy doesn't exist in the latest AWS worker node AMI, but the userdata teamplate try to restart it, that will encounter error as below
Failed to restart kube-proxy.service: Unit not found.
Remove the kube-proxy from restart step
On how to define nodeSelector with autoscaling?
pods can be deployed to any nodes.
I can manually set nodeSelector with label command to several nodes, but in autoscaling environment, how to work it out?
I found there are codes about worker_groups, but not sure how to use for labelling.
# the commented out worker group list below shows an example of how to define
# multiple worker groups of differing configurations
# worker_groups = "${list(
# map("asg_desired_capacity", "2",
# "asg_max_size", "10",
# "asg_min_size", "2",
# "instance_type", "m4.xlarge",
# "name", "worker_group_a",
# ),
# map("asg_desired_capacity", "1",
# "asg_max_size", "5",
# "asg_min_size", "1",
# "instance_type", "m4.2xlarge",
# "name", "worker_group_b",
# ),
# )}"
Terraform apply fails with
* module.eks.null_resource.update_config_map_aws_auth: Error running command 'kubectl apply -f ./config-map-aws-auth_beam-eks.yaml --kubeconfig ./kubeconfig_beam-eks': exit status 1. Output: error: unable to recognize "./config-map-aws-auth_beam-eks.yaml": Unauthorized
This is my configuration for the eks module.
I have a really basic vpc created via terraform-aws-modules/vpc/aws.
module "eks" {
source = "terraform-aws-modules/eks/aws"
cluster_name = "beam-eks"
subnets = "${module.vpc.public_subnets}"
vpc_id = "${module.vpc.vpc_id}"
}Apply succeeds
I need thank for the hide feature that I can manage multiple worker groups in one EKS, test version is v1.3.0.
So below codes work properly.
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/eks_test_fixture/main.tf#L19-L34
My use case is, I need manage two groups of nodes, one for applications, one for monitoring service only. Later will add more node groups (label for nodeSelector ) for different purpose.
Should we uncomment these lines or make another test cases?
A couple of changes would be it easier to work with multiple clusters.
Tags currently are too prescriptive. I have a use case where I need to tag different ASGs with different tags. Im using the ability to push these tags down to node labels and taints to drive different workloads on my kubernetes cluster. Atm, it seems that I can only define tags once on the top level EKS module and these tags are used globally throughout. I would like to be able to define tags per ASGs. A sensible place to provide these seems to be the list of worker_groups maps.
Tags are defined once in var.tags and used throughout both to tag the cluster resources itself, as well as, all ASGs that are created.
Should be able to provide tags in the list of worker_groups maps and these should be used to tagging the corresponding ASGs that are created for each respective worker group. If the tags are not set, they can default to the existing top-level global tag variable that is used.
I can submit a PR if this is reasonable.
The new amazon-eks-node-* AMI with bootstrap script has been released. However, it's not backward compatible with the old AMI and doesn't work with this module.
https://aws.amazon.com/blogs/opensource/improvements-eks-worker-node-provisioning/
This module only works with the eks-woker-* AMIs.
N/A
This module should also work with the new amazon-eks-node-* AMI. The entire userdata.sh.tpl can be reduced to something like this:
# Allow user supplied pre userdata code
${pre_userdata}
# Bootstrap and join the cluster
/etc/eks/bootstrap.sh --b64-cluster-ca '${cluster_auth_base64}' --apiserver-endpoint '${endpoint}' --kubelet-extra-args '${kubelet_extra_args}' '${cluster_name}'
# Allow user supplied userdata code
${additional_userdata}I can contribute, but would like to discuss on how we want to approach backward compatibility first.
See:
This is a general question about the direction of this module.
We get requests that would require this module to manage or create Kubernetes resources. Some examples:
I think we should have a clear position on this types of issues.
The EKS plugin currently supports being able to pass in 1 cluster and worker security group by id.
I think it would make sense to support specifying an array of security group ids.
We have a use case where we need to attach multiple security groups. Some of which are predefined.
Right now in the user-data script we have
sed -i s,MAX_PODS,20,g /etc/systemd/system/kubelet.serviceThe value 20 is hardcoded right now. Since AWS released the numbers in their CloudFormation template, I think we can extract the value and use a lookup function to get the proper value.
A proposal:
locals {
# Mapping from the node type that we selected and the max number of pods that it can run
# Taken from https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-06-05/amazon-eks-nodegroup.yaml
max_pod_per_node = {
c4.large = 29
c4.xlarge = 58
c4.2xlarge = 58
c4.4xlarge = 234
c4.8xlarge = 234
c5.large = 29
c5.xlarge = 58
c5.2xlarge = 58
c5.4xlarge = 234
c5.9xlarge = 234
c5.18xlarge = 737
i3.large = 29
i3.xlarge = 58
i3.2xlarge = 58
i3.4xlarge = 234
i3.8xlarge = 234
i3.16xlarge = 737
m3.medium = 12
m3.large = 29
m3.xlarge = 58
m3.2xlarge = 118
m4.large = 20
m4.xlarge = 58
m4.2xlarge = 58
m4.4xlarge = 234
m4.10xlarge = 234
m5.large = 29
m5.xlarge = 58
m5.2xlarge = 58
m5.4xlarge = 234
m5.12xlarge = 234
m5.24xlarge = 737
p2.xlarge = 58
p2.8xlarge = 234
p2.16xlarge = 234
p3.2xlarge = 58
p3.8xlarge = 234
p3.16xlarge = 234
r3.xlarge = 58
r3.2xlarge = 58
r3.4xlarge = 234
r3.8xlarge = 234
r4.large = 29
r4.xlarge = 58
r4.2xlarge = 58
r4.4xlarge = 234
r4.8xlarge = 234
r4.16xlarge = 737
t2.small = 8
t2.medium = 17
t2.large = 35
t2.xlarge = 44
t2.2xlarge = 44
x1.16xlarge = 234
x1.32xlarge = 234
}
workers_userdata = <<USERDATA
#!/bin/bash -xe
CA_CERTIFICATE_DIRECTORY=/etc/kubernetes/pki
CA_CERTIFICATE_FILE_PATH=$CA_CERTIFICATE_DIRECTORY/ca.crt
mkdir -p $CA_CERTIFICATE_DIRECTORY
echo "${aws_eks_cluster.this.certificate_authority.0.data}" | base64 -d > $CA_CERTIFICATE_FILE_PATH
INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
sed -i s,MASTER_ENDPOINT,${aws_eks_cluster.this.endpoint},g /var/lib/kubelet/kubeconfig
sed -i s,CLUSTER_NAME,${var.cluster_name},g /var/lib/kubelet/kubeconfig
sed -i s,REGION,${data.aws_region.current.name},g /etc/systemd/system/kubelet.service
sed -i s,MAX_PODS,${lookup(local.max_pod_per_node, var. workers_instance_type)},g /etc/systemd/system/kubelet.service
sed -i s,MASTER_ENDPOINT,${aws_eks_cluster.this.endpoint},g /etc/systemd/system/kubelet.service
sed -i s,INTERNAL_IP,$INTERNAL_IP,g /etc/systemd/system/kubelet.service
DNS_CLUSTER_IP=10.100.0.10
if [[ $INTERNAL_IP == 10.* ]] ; then DNS_CLUSTER_IP=172.20.0.10; fi
sed -i s,DNS_CLUSTER_IP,$DNS_CLUSTER_IP,g /etc/systemd/system/kubelet.service
sed -i s,CERTIFICATE_AUTHORITY_FILE,$CA_CERTIFICATE_FILE_PATH,g /var/lib/kubelet/kubeconfig
sed -i s,CLIENT_CA_FILE,$CA_CERTIFICATE_FILE_PATH,g /etc/systemd/system/kubelet.service
systemctl daemon-reload
systemctl restart kubelet kube-proxy
USERDATA
}@brandoconnor Please let me know if this is OK, I'll create a fork and a pull request later
Worker node number and instance type cannot be configured.
Configuration options for worker node number and instance type can be specified in module inputs.
Currently we have to add the policy outside this module but I think 90% of people will use the cluster-autoscaler so it would be cool to have it included in this module and perhaps enabled with a variable.
kops currently has this by default here.
The policy would look something like this:
data "aws_iam_policy_document" "eks_node_autoscaling" {
statement {
sid = "eksDemoNodeAll"
effect = "Allow"
actions = [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:GetAsgForInstance",
]
resources = ["*"]
}
statement {
sid = "eksDemoNodeOwn"
effect = "Allow"
actions = [
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"autoscaling:UpdateAutoScalingGroup",
]
resources = ["*"]
condition {
test = "StringEquals"
variable = "autoscaling:ResourceTag/Name"
values = ["xxxx-eks_asg"]
}
}
}This allows would allow the cluster-autoscaler the access it needs to run correctly.
What do you think?
Currently, in workers.tf, we have this security group:
resource "aws_security_group_rule" "workers_ingress_cluster" {
description = "Allow workers Kubelets and pods to receive communication from the cluster control plane."
protocol = "tcp"
security_group_id = "${aws_security_group.workers.id}"
source_security_group_id = "${local.cluster_security_group_id}"
from_port = 1025
to_port = 65535
type = "ingress"
count = "${var.worker_security_group_id == "" ? 1 : 0}"
}Basically, this setting makes it impossible for Kubernetes services to access pods that have have containerPort set to anything below 1025, which is a huge issue since so many of them use the 80 port (e.g. nginx). So, from_port should be set to 0, not 1025.
I realize this is copied from CloudFormation in the official EKS guide, so I'll also submit an issue there.
I would prefer to use the default security groups created for the cluster, but do not want the default API/32 to be used.
Currently, if you use the default security groups it will create a security group role that allows communication with the eks cluster over the current API/32 cidr.
I want to override the API/32 cidr and specify my own.
terraform plan produces this output when any worker group parameters are computed values:
laverya:~/dev$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.http.workstation_external_ip: Refreshing state...
data.aws_region.current: Refreshing state...
data.aws_availability_zones.available: Refreshing state...
data.aws_iam_policy_document.cluster_assume_role_policy: Refreshing state...
data.aws_iam_policy_document.workers_assume_role_policy: Refreshing state...
data.aws_ami.eks_worker: Refreshing state...
Error: Error refreshing state: 1 error(s) occurred:
* module.eks.data.template_file.userdata: data.template_file.userdata: value of 'count' cannot be computed
provider "aws" {
version = "~> 1.27"
region = "us-east-1"
}
data "aws_availability_zones" "available" {}
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "1.37.0"
name = "eks-vpc"
cidr = "10.0.0.0/16"
azs = ["${data.aws_availability_zones.available.names[0]}", "${data.aws_availability_zones.available.names[1]}", "${data.aws_availability_zones.available.names[2]}"]
private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
public_subnets = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
tags = "${map("kubernetes.io/cluster/terraform-eks", "shared")}"
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "1.3.0"
cluster_name = "terraform-eks"
subnets = ["${module.vpc.private_subnets}", "${module.vpc.public_subnets}"]
tags = "${map("Environment", "test")}"
vpc_id = "${module.vpc.vpc_id}"
worker_groups = [
{
name = "default-m5-large"
instance_type = "m5.large"
subnets = ""
# subnets = "${join(",", module.vpc.private_subnets)}"
},
]
}
Uncomment subnets = "${join(",", module.vpc.private_subnets)}" to replace subnets = "" in the worker_groups config and run terraform plan.
terraform plan completes and a plan is produced.
I have not yet identified the root cause.
This makes it rather difficult to assign subnets to worker groups.
I created EKS cluster in private subnets, we also have discussed about this topic in several tickets, we agree to create EKS workers in private subnets only.
Now it's time to decide, should we keep the feature to assign public IP to EKS workers?
If it is not required any more, I will raise PR to remove this line directly. Otherwise, I have to update with a condition, which way you like?
resource "aws_launch_configuration" "workers" {
name_prefix = "${var.cluster_name}-${lookup(var.worker_groups[count.index], "name", count.index)}"
- associate_public_ip_address = "${lookup(var.worker_groups[count.index], "public_ip", lookup(var.workers_group_defaults, "public_ip"))}"
security_groups = ["${local.worker_security_group_id}"]
iam_instance_profile = "${aws_iam_instance_profile.workers.id}"
image_id = "${lookup(var.worker_groups[count.index], "ami_id", data.aws_ami.eks_worker.id)}"
instance_type = "${lookup(var.worker_groups[count.index], "instance_type", lookup(var.workers_group_defaults, "instance_type"))}"When create workers in private subnets, public IPs are assigned to these workers.
Yes, I will
Provisioning EKS cluster in new AWS account will result in an error when attempting to provision a load balancer if no load balancers of any kind have been provisioned before.
No previous load balancers ( i.e. service-link role AWSServiceRoleForElasticLoadBalancing doesn't exist)
AccessDenied: User: <MODULE-PROVISIONED-ROLE> is not authorized to perform: iam:CreateServiceLinkedRole on resource: arn:aws:iam::<ACCOUNT-ID>:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing
because EKS is attempting to create the ELB service-link role for you, and the roles created by the module lack iam:CreateServiceLinkedRole
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.15.2
ports:
- containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
name: nginxservice
spec:
type : LoadBalancer
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
EKS should provision load balancer.
Module should optionally provision (via flag) a resource "aws_iam_service_linked_role", or include updated IAM policies (iam:CreateServiceLinkedRole) to allow the EKS cluster to provision the required service-link role. Alternatively, if this is deemed not the responsibility of the module, the "Assumptions" section in README.md should note the issue.
Possibly, depending on the choice of solution (implementation change, documentation update)
AWS Service Link FAQ:
https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/elb-service-linked-roles.html#create-service-linked-role
How is cluster DNS supposed to work? I have not been able to get pods to resolve any cluster addresses (including kubernetes.default) using EKS. I suspect it's a function of how the AWS VPC CNI works (or doesn't) and figured other people using this module must be running into the same problem however I can't seem to find much on the internet about this in EKS.
locals {
worker_groups = "${list(
map(
"name", "k8s-worker",
"ami_id", "ami-73a6e20b",
"asg_desired_capacity", "5",
"asg_max_size", "8",
"asg_min_size", "5",
"instance_type","m4.large",
"key_name", "${aws_key_pair.infra-deployer.key_name}"
),
)}"
tags = "${map("Environment", "${terraform.workspace}")}"
}
data "aws_vpc" "vpc" {
filter {
name = "tag:env"
values = ["${terraform.workspace}"]
}
filter {
name = "tag:Name"
values = ["${terraform.workspace}-us-west-2"]
}
}
data "aws_subnet_ids" "eks_subnets" {
vpc_id = "${data.aws_vpc.vpc.id}"
tags {
env = "${terraform.workspace}"
Name = "${terraform.workspace}-eks*"
}
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
cluster_name = "${terraform.workspace}"
subnets = "${data.aws_subnet_ids.eks_subnets.ids}"
vpc_id = "${data.aws_vpc.vpc.id}"
kubeconfig_aws_authenticator_env_variables = "${map("AWS_PROFILE", "infra-deployer" )}"
map_accounts = ["${lookup(var.aws_account_ids, "prod")}"]
worker_groups = "${local.worker_groups}"
tags = "${local.tags}"
}
Trying DNS on a brand new cluster:
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 172.20.0.10
Address: 172.20.0.10:53
** server can't find kubernetes.default: NXDOMAIN
*** Can't find kubernetes.default: No answer
$ kubectl exec -ti busybox -- cat /etc/resolv.conf
nameserver 172.20.0.10
search default.svc.cluster.local svc.cluster.local cluster.local staging.thinklumo.com us-west-2.compute.internal
options ndots:5
I've tried too many things to list here and at this point suspect its an issue with EKS so I'm hoping someone has been down this path already.
N/A
Terraform v0.11.7
+ provider.aws v1.25.0
I want to be able to run additional user data before the plugin user data on the worker launch config.
I am behind a proxy and need to configure the proxy information before anything else happens.
The plugin only provides a way to specify additional user data that runs after the plugins user data.
The AWS CNI by default pre-allocates the max number of IPs per node which results in unnecessary depletion of my IP pool. As of CNI 1.1, you can fix this by setting WARM_IP_TARGET in the aws-k8s-cni.yaml but this needs to be applied before the EC2 instances are created.
Is there a way I can have Terraform apply a k8s config between creating the cluster and creating the worker groups? My current workaround is specifying 0 nodes in the Terraform module, applying my custom aws-k8s-cni.yaml, then changing the worker node count to my actual desired number.
Thanks!
The cluster is created using the release aws-k8s-cni.yaml which does not have WARM_IP_TARGET set.
The reason is that after cluster creation, almost everyone will run the k8s node autoscaler. This autoscaler is changing the desired_capacity to suit resources required by the cluster. So when the cluster autoscales, then TF is run again later, you see something like this:
~ module.cluster_1.aws_autoscaling_group.workers
desired_capacity: "5" => "3"
You can just add a lifecycle statement to resource aws_autoscaling_group.workers:
lifecycle {
ignore_changes = [ "desired_capacity" ]
}
For my current delivery, the customer has credentials with multiple profiles and not only needs to specify different profiles per cluster, but has no default profile.
It would be great if the kubeconfig.tpl could be modified:
...
users:
- name: aws
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: heptio-authenticator-aws
args:
- "token"
- "-i"
- "${cluster_name}"
env:
- name: AWS_PROFILE
value: ${aws_profile}where the default Terraform value to populate the template would be default to ensure no regression.
I wanted to start a discussion before a PR to ensure best path forward on this. Thanks!
I would be nice to have an option which allows to install addons together with the cluster. There is another Terraform Registry module which does that: https://registry.terraform.io/modules/scholzj/kubernetes/aws/1.3.4#addons
After deployingeks via this TF module in a brand new AWS account, the internet-facing k8s service I created could not create a load balancer. Turns out it's because this is a brand new AWS account and no ELB has been created in it before and the AWS user guide (as well as this module) assumes that AWSServiceRoleForElasticLoadBalancing already exists.
Recommend adding
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeAccountAttributes"
],
"Resource": "*"
}
]
}
To the cluster role policy.
In the getting started example
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "1.14.0"
name = "test-vpc"
cidr = "10.0.0.0/16"
azs = ["${data.aws_availability_zones.available.names[0]}", "${data.aws_availability_zones.available.names[1]}", "${data.aws_availability_zones.available.names[2]}"]
private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
public_subnets = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
enable_nat_gateway = true
single_nat_gateway = true
tags = "${merge(local.tags, map("kubernetes.io/cluster/${local.cluster_name}", "shared"))}"
}
module "eks" {
source = "../.."
cluster_name = "${local.cluster_name}"
subnets = ["${module.vpc.public_subnets}", "${module.vpc.private_subnets}"]
tags = "${local.tags}"
vpc_id = "${module.vpc.vpc_id}"
worker_groups = "${local.worker_groups}"
worker_group_count = "1"
map_roles = "${var.map_roles}"
map_users = "${var.map_users}"
map_accounts = "${var.map_accounts}"
}
Both private and public subnets are passed to eks module as a single variable. How does eks module determine which subnets are public and which subnets are private and thus launch worker into private subnets only?
Recently I upgrade release from 1.1.0 to 1.3.0 and add some changes to launch configuration, such as key name,
* aws_launch_configuration.workers (deposed #0): ResourceInUse: Cannot delete launch configuration project-prod-020180630105107074900000001 because it is attached to AutoScalingGroup project-prod-monitoring
status code: 400, request id: 46f32656-8661-11e8-9e77-51ef4818b760
change ami image id, add/remove key pair name, or other which need re-create a new launch configuration.
Smoothly updated.
I am still investigating this issue, if I can fix, will raise PR.
Here is the fix someone mentioned:
There is no mention of the requirement to have kubectl installed before running the script. The module will fail while applying the plan.
Error: Error applying plan:
1 error(s) occurred:
* module.eks.null_resource.configure_kubectl: Error running command 'kubectl apply -f .//config-map-aws-auth.yaml --kubeconfig .//kubeconfig': exit status 127. Output: /bin/sh: 1: kubectl: not found
Have the a note and a link to the install instructions in the Assumption section of the README.md.
If region is set to us-west-1:
Error: Error refreshing state: 1 error(s) occurred:
module.eks.data.aws_ami.eks_worker: 1 error(s) occurred:
module.eks.data.aws_ami.eks_worker: data.aws_ami.eks_worker: Your query returned no results. Please change your search criteria and try again.
module "eks" {
source = "terraform-aws-modules/eks/aws"
cluster_name = "test-eks-cluster"
vpc_id = "${module.vpc.default_vpc_id}"
subnets = "${module.vpc.public_subnets}"
tags = {
Environment = "test"
Terraform = "true"
}
}
An available AWS AMI ID.
Not sure how.
[ * ] bug report
The generated config-map-aws-auth***.yaml and kubeconfig.yaml file is saved to the root folder, the file name is appended with the value of the config_output_path variable
Create a cluster with a custom config_output_path
The config-map-aws-auth***.yaml file should be saved to the config_output_path
aws_auth.tf
line 3 - missing /
filename = "${var.config_output_path}/config-map-aws-auth_${var.cluster_name}.yaml"
line 9 - missing /
command = "kubectl apply -f ${var.config_output_path}/config-map-aws-auth_${var.cluster_name}.yaml --kubeconfig ${var.config_output_path}/kubeconfig_${var.cluster_name}"
kubectl.tf
line 3 - missing /
filename = "${var.config_output_path}/kubeconfig_${var.cluster_name}"
running EKS cluster for a week, get disk space issue, need feature to control and extend with larger size.
No control on root block device, disk space is used out quickly
The root_block_device mapping supports the following:
volume_type - (Optional) The type of volume. Can be "standard", "gp2", or "io1". (Default: "standard").
volume_size - (Optional) The size of the volume in gigabytes.
iops - (Optional) The amount of provisioned IOPS. This must be set with a volume_type of "io1".
delete_on_termination - (Optional) Whether the volume should be destroyed on instance termination (Default: true).
Currently, only enable delete_on_termination
Yes, I will
Worker ASG ARNs are exposed, but not names. ASG names are used in aws_autoscaling_attachment among other things.
Both are exposed
PR: #77
I have difficulty to troubleshooting EKS nodes issue, for example, OutOfDisk issue. I go to cloudwatch, there is no any instance logs or /var/log/message logs.
OutOfDisk Unknown Fri, 13 Jul 2018 00:11:01 +0000 Fri, 13 Jul 2018 00:11:43 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Currently there is no key pair set and I can't login the eks nodes to do further check.
No any logs from EKS cluster.
I need some ways to review the logs when something happened.
not sure how to fix this issue, need help.
Amazon's EKS access control is managed via the aws-auth configmap which allows multiple IAM users and roles (cross-account capable) to be granted group membership. The current implementation only allows worker node access, this should be configurable to allow more access control rules to be specified per the documentation: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
The current implementation only allows worker node access.
Ability to specify role/user/account mappings for group membership.
Please Help! I ran everything with defaults other then setting the VPC and Subnet
I get this error running the Terraform apply
Error: Error applying plan:
1 error(s) occurred:
Set the config on the instance
No can I please have help?
First time setting this up!
It's not possible to define what the Name tag of any resource will absolutely be. This should be able to be user defined.
I think another variable map containing tag_defaults or a local (since computing is needed) variable will come in handy here. Will explore in next week's cycles.
I have a need to to taint nodes and did a quick implementation to support passing taints to the register-with-taints kubelet flag. Take a look here: perryao@435f62e.
But figured it might be worth talking through a solution that supports more of the flags without the need for one-off PRs for each one.
What do you think?
Flag docs here: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
This is much more a question than an issue. I see the workstation cidr being allowed to access 443 in the security group attached to the eks cluster. I see the same thing in the terraform eks getting started post. My assumption was this would limit access to the kubernetes api (control plane) to that cidr. It doesn't do that and these control planes are fully accessible on the internet. Was the intention to allow only that cidr to access the control plane? I really wish that were possible. I'm likely completely missing the reason to allow that ingress.
I expected the control plane to be limited to the IP address in the cidr. My expectation may be completely wrong in which case maybe a different variable description could help.
Currently, in order to achieve blue/green deployment with worker groups (i.e. updating to a new AMI), I have to add a new worker group with the updated AMI, let them spin up, drain the old nodes so pods transition, then scale down the old worker group (set min/max/desired to 0).
This is not a terrible way of doing it but the problem is that the old ASG (and related resources) sticks around forever and there doesn't seem to be a way to clean up the old stuff without major surgery. If I change the AMI 3 times, I now have 3 worker groups - 2 inactive and scaled to 0 and one active.
Is there a better way of doing this with this module? There's a distinct possibility I'm missing some fundamental terraform concepts but this seems like a complex issue to me. My code ends up looking like this after a new worker group is fully deployed and the old is scaled down (you can see how even semi-frequent deployments would make this list long and leave a lot of trailing garbage):
map(
"name", "k8s-worker-179fc16f",
"ami_id", "ami-179fc16f",
"asg_desired_capacity", "0",
"asg_max_size", "0",
"asg_min_size", "0",
),
map(
"name", "k8s-worker-67a0841f",
"ami_id", "ami-67a0841f",
"asg_desired_capacity", "5",
"asg_max_size", "8",
"asg_min_size", "5",
"instance_type","${lookup(var.worker_sizes, "${terraform.workspace}")}",
"key_name", "${aws_key_pair.infra-deployer.key_name}",
"root_volume_size", "48"
)
I have seen other code around the internet that does blue/green ASGs but those are for much simpler use-cases IMO - a create_before_destroy and letting it rip would bring a K8s cluster down. I have no qualms with multiple apply steps - its the cleanup part that I'm after.
Currently the module only supports creation of the security groups for the cluster and workers from within the module itself. Some of the rules are 100% necessary and others are just commonplace and therefore useful. The rules rely on dynamic values but could be applied just the same to a security group passed to the module instead of created within the module. This would give flexibility to the module consumer to provide their own security group and a predefined set of rules that might be tighter than what the module currently prescribes.
NA
The module should be able to accept a security group ID as input for both the cluster and workers with rules defined outside the module.
I want to access the worker nodes via SSH using a Key Pair (Private-Public Key). However there is no option which allows me to specify a key pair to be installed on the worker nodes.
Provide an option to specify a key pair (local or already in AWS) and installed it on the worker nodes. It should be possible to set the key name via variable. For inspiration have a look at Terraform-DC/OS module: https://github.com/dcos/terraform-dcos/tree/master/aws#configure-aws-ssh-keys
It seems the cluster it is running with Authorization enabled (like RBAC) and there is no permissions for the ingress controller. Please check the configuration
I can't deploy alb-ingress-controller with the above error.
I am not 100% sure it is a bug.
After create EKS cluster with this module, I went through the steps to step 4, I got this error.
This step has additional help, but not sure how to set in EKS cluster with this module.
Deploy the modified alb-ingress-controller.
$ kubectl apply -f alb-ingress-controller.yaml
The manifest above will deploy the controller to the kube-system namespace. If you deploy it outside of kube-system and are using RBAC, you may need to adjust RBAC roles and bindings.
Should work without error.
I want to use the Ubuntu EKS image (custom) AMI and my settings are ignored by the module.
A custom AMI ID is ignored by the module. I tried to set the AMI ID to the following ami_id in the workers_group_defaults section:
workers_group_defaults = {
...
ami_id = "ami-39397a46" # Ubuntu Image
...
}
The specified AMI is the Ubuntu EKS image for us-east-1. Ubuntu released a statement that they will support and update an image specifically for EKS. The image AMI ID's can be found here: https://cloud-images.ubuntu.com/aws-eks/?_ga=2.56651242.1343651116.1533683680-508754220.1533683680
I would like to use the Ubuntu image instead of the Amazon AMI to make my environment more portable.
Use try to set the ami_id in the workers_group_defaults to use the Ubuntu EKS AMI ID ami-39397a46 (us-east-1) or ami-6d622015 (us-west-2).
Custom AMI ID's can be used for worker nodes. Specifically, Ubuntu EKS can be used with this module.
Maybe this is just a misunderstanding or it is super easy to fix. If not let me know.
Based on this guide from aws, it is recommended that you specify both public and private subnets when creating your eks cluster, but that you only create your worker nodes in your private subnets. Current behaviour in this module will use the same subnets for creating the eks cluster as for placing the worker nodes within.
I believe it would be a good feature to add an additional (optional) list variable to the module called worker_subnets that will be used to create the worker nodes within. This means you can add private and public subnets to the subnets variable, but only add private subnets to the worker_subnets variable.
I have a branch with this feature on a fork, I will add a PR to be looked at.
Currently we have:
resource "aws_iam_role" "workers" {
name_prefix = "${aws_eks_cluster.this.name}"
assume_role_policy = "${data.aws_iam_policy_document.workers_assume_role_policy.json}"
}
resource "aws_iam_instance_profile" "workers" {
name_prefix = "${aws_eks_cluster.this.name}"
role = "${aws_iam_role.workers.name}"
}Is there a reason to use name_prefix instead of just name? I ask because the resultant names are things like my-cluster-20180808095045107900000005.
We have to create cross account IAM policies for things like ECR and it would be nice to have a predictable and consistent name for the roles 🙂
Although worker nodes are deployed as autoscaling group, when EKS cannot schedule more pods because of missing resources i.e. CPU more nodes are not started by ASG. Would be nice to deploy Cluster Autoscaler automatically (or at least put some information in README how to do this) so we can benefit from ASG.
ASG workers are not started even if EKS cannot schedule more pods because of missing CPU and we are still below the maximum size of workers ASG.
The expected behaviour would be:
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
)
2. Scale sample application so it needs more than whole CPU from single VM.
3. See that Autoscaler is adding more nodes.
ESK in us-east-1
v0.11.7
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.