!! NOT FOR PRODUCTION USE !!

This is just a challenge to study and practice EKS deployment using Terraform.

• Create an EKS cluster using at least two Availability Zones for worker nodes
• Deploy a Prometheus instance running on EKS
• Deploy a network monitoring solution for Kubernetes
• Validate communication between nodes
• Extract node latency communication metrics from Prometheus

• Terraform v0.14.7
• Configured AWS CLI
• AWS IAM Authenticator
• kubectl v1.19.8
• wget (required for the eks module)

Recommendation: Use asdf and .tools-versions file for version control.

• vpc.tf provisions a VPC, subnets and availability zones using the AWS VPC Module.
• security-groups.tf provisions the security groups used by the EKS cluster.
• eks-cluster.tf Use EKS module to provision all the resources required to set up the cluster.
• outputs.tf defines the output configuration.
• versions.tf sets the Terraform version to at least 0.14.
• kubernetes.tf includes kubernetes module to manage kubernetes_config_map.aws_auth
• metrics-server.tf deploy metrics-server on EKS cluster.
• kube-prommetheus.tf use Helm provider to deploy kube-prometheus-stack.
• values-kube-prometheus.yaml custom helm values for running kube-prometheus-stack on EKS.

After the terraform apply, the EKS should be available with two nodes and the monitoring stack deployed.

A kubeconfig file is generated by the EKS module after the install.
The KUBECONFIG env var should be set to configure the kubectl CLI with the EKS context:

export KUBECONFIG=$(terraform output -raw kubectl_config)

Then, kubectl can be used to check that the nodes are Ready and using two different zones:

# kubectl get nodes
NAME                         STATUS   ROLES    AGE    VERSION
ip-10-0-1-94.ec2.internal    Ready    <none>   10m   v1.19.6-eks-49a6c0
ip-10-0-2-204.ec2.internal   Ready    <none>   10m   v1.19.6-eks-49a6c0

The metrics-server is deployed in the kube-system namespace. The kube-prometheus-stack release is deployed in the monitoring namespace:

$ kubectl get po --all-namespaces
NAMESPACE     NAME                                                     READY   STATUS    RESTARTS   AGE
kube-system   aws-node-2p7b4                                           1/1     Running   0          3h19m
kube-system   aws-node-kpqfc                                           1/1     Running   0          3h19m
kube-system   coredns-7d74b564bd-8r8l8                                 1/1     Running   0          3h23m
kube-system   coredns-7d74b564bd-t9kvr                                 1/1     Running   0          3h23m
kube-system   kube-proxy-5ms7q                                         1/1     Running   0          3h19m
kube-system   kube-proxy-q6hdc                                         1/1     Running   0          3h19m
kube-system   metrics-server-7cc47dc976-xdmsb                          1/1     Running   0          3h21m
monitoring    alertmanager-kube-prometheus-kube-prome-alertmanager-0   2/2     Running   0          3h17m
monitoring    kube-prometheus-grafana-5cd7cdd777-jf97t                 2/2     Running   0          3h18m
monitoring    kube-prometheus-kube-prome-operator-7c7cd8f485-h7fbg     1/1     Running   0          3h18m
monitoring    kube-prometheus-kube-state-metrics-5f575558c7-pd74r      1/1     Running   0          3h18m
monitoring    kube-prometheus-prometheus-node-exporter-lcrwb           1/1     Running   0          3h18m
monitoring    kube-prometheus-prometheus-node-exporter-z29hb           1/1     Running   0          3h18m
monitoring    prometheus-kube-prometheus-kube-prome-prometheus-0       2/2     Running   1          3h17m

The kube-prometheus-stack chart was deployed using the Terraform Helm provider.

The kube-prometheus is a stack that is meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components.

Components included in this package are:

• The Prometheus Operator
• Prometheus
• Alertmanager
• Prometheus node-exporter
• Prometheus Adapter for Kubernetes Metrics APIs
• kube-state-metrics
• Grafana

In addition to that, the package delivers a default set of dashboards and alerting rules. Many of the useful dashboards and alerts come from the kubernetes-mixin project.

Prometheus can be accessed with kubectl port-forward:

$ kubectl --namespace=monitoring port-forward svc/kube-prometheus-kube-prome-prometheus 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

Then, go to http://localhost:9090/alerts/ :

All alerts are inactive (except the Watchdog alert, which is intended to be an "always firing" event).

The dashboards can be accessed by proxying on the Grafana deployment:

$ kubectl --namespace=monitoring port-forward deploy/kube-prometheus-grafana 3000:3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Go to http://localhost:3000/ and login with the following default credentials:

admin
prom-operator

Note: The values-kube-prometheus.yaml file can be used to change the default password before installation.

The default dashboards bundled with kube-prometheus already provides insightful data about the cluster compute resources, workload and network metrics.

API Server Work Queue Latency and resources usage:

Kubelet request duration and resources usage:

Cluster Network Metrics overview:

This case study was able to use Terraform to deploy an EKS cluster with the Prometheus monitoring solution.

Reproducibility is guaranteed as all AWS infrastructure dependencies are managed by Terraform. No manual steps were needed.

The .terraform.tfstate file and the Helm releases can handle both infrastructure and applications states for incremental upgrades.

Some resources used in this case study are not included in the AWS Free Tier.

The use of this project will incur costs, especially the EKS cluster (currently $0.10 per hour).

Remember to terraform destroy after use!