getamis / vishwakarma

Terraform modules to create a self-hosting Kubernetes cluster on opinionated Cloud Platform.

Vishwakarma

Vishwakarma can be used to create a Kubernetes cluster in AWS by leveraging HashiCorp Terraform and CoreOS. Of course, we didn't develop it from scratch, we refer to CoreOS Tectonic, before starting to dive into the detail, let's experience it first.

Features

• Kubernetes v1.27.2+.
• Supported AWS VPC CNI, flannel, cilium networking.
• RBAC-enabled, Audit log, and etcd data encryption.
• etcd v3.5.0+.
• On-cluster etcd with TLS.

Requirements

• Terraform: All of the AWS resource will be create by Terraform, hence, you need to install it and confirm the permission setup correctly, then Terraform have the permission to create AWS resource automatically. Minimum required version of Terraform is v1.2.0.

• kubectl: After the cluster created completely, there is a Kubernetes ConfigMap aws-auth need to be created through kubectl, so need to install it. Minimum required version of Kubernetes is v1.27.0.

• aws-iam-authenticator: The clsuter access permission integrate with AWS IAM, in order to let the cluster know whether you have the right to access, aws-iam-authenticator need to be installed in the client side.

• Key Pair: In order to access worker node through ssh protocol, please create a key pair in example region US West (Oregon) us-west-2.

• jq: It's a necessary command-line for filtering JSON in many operations.

• python: In order to support node rolling udate, need the lambda function, hence, this tf module need python 3.7 and python package management tool pip

Getting Started

First, acquire Vishwakarma from github:

$ git clone https://github.com/getamis/vishwakarma.git

Second, before the operation, user need to create a AWS EC2 key pairs first, and input it when there is command line prompt during the operation.

# need to input the key pair name
var.key_pair_name
  The key pair name for access bastion ec2
Enter a value:

Creating a Kubernetes clsuter

Please create a ssh key pair in ~/.ssh/ with the name id_rsa.pub and id_rsa, this example use the key pair for the etcd, Kubernetes master, Kubernetes node EC2 instance (refer to Here for the more detail information).

# switch to kubernetes-cluster example folder
$ cd examples/kubernetes-cluster

# initial for sync terraform module and install provider plugins
$ terraform init

# create the network infrastructure
$ terraform apply -target=module.network

# create the kubernetes master compoment
$ terraform apply -target=module.master

# create the general and spot Kubernetes worker group
$ terraform apply

Verify the Kubernetes cluster is up! (Still keep in the same folder):

# Get the kubeconfig from S3 (The bucket name is demo-elastikube-xxxxxxxx. 
# The prefix demo-elastikube is the cluster name defined in main.tf and the rest part is an MD5.
# setup kubeconfig for kubectl to access Kubernetes cluster
$ export KUBECONFIG=#{The Path You Put kubeconfig}/kubeconfig

# check whether there is 4 worker register successfully, it will takes several minutes...
$ kubectl get node

NAME                          STATUS    ROLES     AGE       VERSION
ip-10-0-48-247.ec2.internal   Ready     master    9m        v1.27.2
ip-10-0-48-117.ec2.internal   Ready     master    9m        v1.27.2
ip-10-0-66-127.ec2.internal   Ready     on-demand 5m        v1.27.2
ip-10-0-66-127.ec2.internal   Ready     on-demand 6m        v1.27.2
ip-10-0-71-121.ec2.internal   Ready     spot      3m        v1.27.2
ip-10-0-86-182.ec2.internal   Ready     spot      4m        v1.27.2

What’s Going On?

You have completed one Kubernetes cluster the same as below picture, and let me briefly explain how Vishwakarma achieves it.

Modules

Vishwakarma includes serveral major modules:

aws/network

Create one AWS VPC including private and public subnet, and one ec2 instance called bastion hosts in public subnet, hence, one can access the resource hosting in the private subnet, refer aws/network for the detail variable inputs.

aws/elastikube

This module creates the Kubernetes control plane, Terraform is responsible for the complicated Kubernetes compoments, and it takes about 10~15 minutes to complete, refer aws/elastikube for the detail variable inputs.

aws/kube-worker

Create a AWS auto-scaling group with CoreOS container linux and leverage ignition to provision and register to ElastiKube automatically.

Due to using AWS launch template, hence, it's up to user to choose spot or on demand instance type by changing the variable, refer aws/kube-worker for the detail variable inputs.

Contributing

There are several ways to contribute to this project:

1. Find bug: create an issue in our Github issue tracker.
2. Fix a bug: check our issue tracker, leave comments and send a pull request to us to fix a bug.
3. Make new feature: leave your idea in the issue tracker and discuss with us then send a pull request!

Changelog

The Changelog captures all important release notes.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

vishwakarma's Issues

Add support for provisioning AKS

Is this a BUG REPORT or FEATURE REQUEST?:
This is a feature request.

What happened:
We want vishwakarma support AKS

What you expected to happen:
Use vishwakarma to provision AKS

How to reproduce it (as minimally and precisely as possible):
N/A

Anything else we need to know?:
N/A

Add support for other cloud platforms

Is this a BUG REPORT or FEATURE REQUEST?:
Feature request

What happened:
Support other cloud platforms, such as Azure, GCP.

What you expected to happen:
Can deploy Kubernetes on other cloud platforms.

ASG Name tags duplicated on modules/aws/kube-worker when give Name tag from extra_tags

Is this a BUG REPORT or FEATURE REQUEST?:
Bug report

What happened:

          + {
              + "key"                 = "Name"
              + "propagate_at_launch" = "true"
              + "value"               = "test-getamis-k8s"
            },
          + {
              + "key"                 = "Name"
              + "propagate_at_launch" = "true"
              + "value"               = "test-getamis-k8s-worker-spot"
            },

What you expected to happen:
Not to use the Name tag from extra_tags, set Name tag by module name variable.
Just like the modules/aws/kube-etcd:

vishwakarma/modules/aws/kube-etcd/main.tf

Lines 78 to 82 in 0631034

	tags = merge(var.extra_tags, map(
	"Name", "${var.name}-etcd-${count.index}",
	"kubernetes.io/cluster/${var.name}", "owned",
	"Role", "etcd"
	))

How to reproduce it (as minimally and precisely as possible):
Launch example cluster:
https://github.com/getamis/vishwakarma/tree/master/examples/kubernetes-cluster

Anything else we need to know?:
Fix in PR: #111

Add support for other Linux distro

Is this a BUG REPORT or FEATURE REQUEST?:
Feature Request

What happened:
Support other Linux distributions.

What you expected to happen:
Can deploy Kubernetes on other Linux distro

Anything else we need to know?:
On other Linux distro, we need to use other tool for provisioning Kubernetes cluster.

discuss: why move away from coreos for AWS EKS?

Hey @smalltown. I'm curious about this commit - 6e6519f. Why did you move away from CoreOS for eks worker nodes? Did you run into issues related to performace? stability?

I'm curious because we were planning to move to coreos (all-in-all) to reduce the host based attack surface.

upgrade k8s cluster

Hi

Deployment of cluster using image quay.io/coreos/hyperkube with version v1.10.5 works well..as soon as we change it to use k8s.gcr.io/hyperkube:v1.10.11 on all the components (worker, master etc..) , cluster broke ..we see etcd running , but api server is not getting started

is this a known issue ?? or what's the procedure in case to upgrade the existing k8s version ?/

thanks
shan

Worker group parameter about role need to align with Self-Hosted

Currently the self-hosted worker iam role parameter name is role_name, that in eks worker is worker_iam_role, I will modify worker_iam_role to role_name, keeping they are consistent

discuss: why move away from coreos for AWS EKS?

Hey @smalltown. I'm curious about this commit - 6e6519f. Why did you move away from CoreOS for eks worker nodes? Did you run into issues related to performace? stability?

I'm curious because we were planning to move to coreos (all-in-all) to reduce the host based attack surface.

Update scripts to support terraform v0.12.x

Add jq command on README Requirements

Is this a BUG REPORT or FEATURE REQUEST?:
Feature request

What happened:
Error message
'go run main.go -key $PKCS_KEY | jq '.keys += [.keys[0]] | .keys[1].kid = ""' > ./examples/iam-auth-and-irsa/k8s-cluster/.secret/keys.json': exit status 127. Output: /bin/sh: jq: command not found

What you expected to happen:
Add jq command on README Requirements

How to reproduce it (as minimally and precisely as possible):
terraform apply -target=module.irsa without jq installed

Anything else we need to know?:
I can help this issue 👋

AWS EKS network issues

I'm experiencing issues with the AWS EKS cluster.

When I run helm init tiller is being installed in the cluster but when I run helm ls I'm getting

Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp 10.0.80.217:10250: getsockopt: no route to host

10.0.80.217 is the secondary private IP for one of the spot instances belonging to the cluster. I do not see that IP being associated with the interface on that node whatsoever, which kind of explains why I'm getting no route to host.

If I assign that IP to the eth0 then the address becomes routable but helm ls then produces different error:

Error: forwarding ports: error upgrading connection: error dialing backend: x509: certificate is valid for 10.0.92.233, not 10.0.80.217

where 10.0.92.233 is the primary private IP for that node.

I was not able to determine what code generates and associates these secondary private IPs with the nodes? I've tried disassociating them from the nodes, but they keep coming back (each time different IP addresses from the private subnet are being generated and associated). And I haven't found a way to control this via terraform so some additional insight on how these are being generated is appreciated.

And any ideas on how to troubleshoot this? I haven't made any changes to the code but am running it as it is. kubectl cmd works as expected, no issues there except for when running kubectl logs... that throws the same type of error.

Ideas are appreciated, thanks!

Migrate CoreOS Container Linux to Fedora CoreOS

Is this a BUG REPORT or FEATURE REQUEST?:
Feature Request

What happened:
Migrate CoreOS Container Linux to Fedora CoreOS, because CoreOS Container Linux has beed deprecated.

What you expected to happen:
Can deploy Kubernetes cluster on Fedora CoreOS.

Anything else we need to know?:
Ignition provider need to use community-terraform-providers/terraform-provider-ignition.

permissions iam

Hello
what permissions are required for the IAM user
Thank you

Add support for provisioning GKE

Is this a BUG REPORT or FEATURE REQUEST?:
FEATURE REQUEST

What happened:
Vishwakarma doesn't support for provisioning GKE cluster.

What you expected to happen:
Make Vishwakarma can provision GKE cluster by Terraform modules.

How to reproduce it (as minimally and precisely as possible):
N/A

Anything else we need to know?:
N/A

cc @smalltown

The irsa s3_object always have to be updated

Is this a BUG REPORT or FEATURE REQUEST?:
Feature request

What happened:
The module.irsa.aws_s3_bucket_object.keys_json always have to be updated, since keys_json is reading from file data source.

Terraform will perform the following actions:

  # module.irsa.data.local_file.keys_json will be read during apply
  # (config refers to values not yet known)
 <= data "local_file" "keys_json"  {
 ...

  # module.irsa.aws_s3_bucket_object.keys_json will be updated in-place
  ~ resource "aws_s3_bucket_object" "keys_json" {
  ...

According to the data source document, it might be caused by data resource won't be checked before apply phase.
https://www.terraform.io/docs/language/data-sources/index.html#data-resource-dependencies

NOTE: In Terraform 0.12 and earlier, due to the data resource behavior of deferring the read until the apply phase when depending on values that are not yet known...

What you expected to happen:
The S3 object only have to be updated when necessary.

How to reproduce it (as minimally and precisely as possible):
Launch example cluster:
https://github.com/getamis/vishwakarma/tree/master/examples/kubernetes-cluster

Then, apply the module.master many times.

terraform apply -target=module.master

Anything else we need to know?:
I've wrote a mitigation PR for this issue: #110