allefebv / cloud-1

:books: devops - iac

cloud-1

📚 school project to discover cloud environments (provisioning / configuration / deployment)

Overview

This project uses various technologies to deploy a previously realized school project website (camagru) in kubernetes, both locally and in the cloud, on AWS.

Technologies used

• Vagrant
• Terraform
• Ansible
• Kubernetes (Kubeadm, Flannel, Ingress-nginx, Metallb)
• AWS (Route53, RDS, EC2, ELB, S3, ACM, KMS)

Provisionning

local

I wanted to deploy a local k8s cluster that would mimic closely a cloud k8s cluster.
To achieve that, I chose vagrant to spin up vms, 1 for the master, 2 as nodes (workers)

cloud

I chose AWS as it's the most used cloud provider as of today.
I used terraform to provision AWS resources neccessary to deploy a minimal self-managed k8s cluster (2 t2.micro ec2, 1 master, 1 node)

Configuration and Deployment

I used ansible to configure k8s and deploy my app to local and cloud environments, with two main roles and subroles :

• configuration : k8s cluster creation
• deployment: spin up the k8s resources to make the app run

playbook usefull tags

• --skip-tags
  • cloud
  • local
  • first-run
• --tags
  • config
  • deployment
    • db
    • pma
    • dashboard
    • app

cloud-1's Issues

"Rolifier" + "Ansibliser" le déploiement des ressources kubernetes spécifiques à l'application (MySQL / PMA / App / Dashboard)

Switch from vagrant user to a user that will work both in AWS and Vagrant

https://ugurakgul.medium.com/bootstrapping-a-kubernetes-cluster-with-ansible-2d1a1155fcb9

=> Ansible user in NOPASSWD mode

=> Solved by having user as a variable.
Vagrant for vagrant, ubuntu for aws

Make kubeadm config a file instead of cmd line

Make project work from vagrant cluster

Tranform camagru project from docker-compose to K8s resources declaration.

Local k8s cluster initialization

Write ansible file that are modular + usable both on AWS and Vagrant

• configure it with containerd, not docker, as k8s dropped docker support
  https://www.aukfood.fr/kubernetes-sans-docker/

https://github.com/hajdaini/k8s-cluster-vagrant-ansible/blob/master/roles/master/tasks/main.yml

https://github.com/Frewx/ansible-kubernetes-cluster/blob/main/roles/configure-network-rules/tasks/main.yml

https://averagelinuxuser.com/kubernetes_containerd/

https://ugurakgul.medium.com/bootstrapping-a-kubernetes-cluster-with-ansible-2d1a1155fcb9

Networking troobleshooting

Pods on the same node can communicate through IP directly
Pods on different nodes cannot communicate through IP

Same node == bridge
Different node == flannel
=> FLANNEL ISSUE

Was using the default interface, eth0, instead of eth1

automate this in the ansible files

K8s volumes write rights for images on phpo container + mount on nginx container to serve images

Automatic config of php database on container creation / launch

Install Composer + aws sdk in containter DockerFile

Db persistance check + volume type in aws

• images on S3
• db on RDS

ANSIBLE dynamic hosts

https://faun.pub/learning-the-ansible-aws-ec2-dynamic-inventory-plugin-59dd6a929c7f

https://devopscube.com/setup-ansible-aws-dynamic-inventory/#:~:text=Dynamic%20inventory%20is%20an%20ansible,it%20became%20an%20Ansible%20plugin.

check that db host works both locally and cloud

• configmap tagged cloud or local for host

Make ansible deployment totally idempotent

Real SSL Certificate

• ok for app manually

  • ingress-controller config TODO in file definition

• ok for pma

  • but error message

• dashboard not working

• config for cloud only, make it variable local/cloud

Nice to have :

• Php code compatible local and cloud
• SSL between DB <-> app/pma
• kubeadm config in file instead of cmd line
• Dashboard in cloud

Ingress + domain name

Bare Metal :
https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal

https://particule.io/en/blog/k8s-no-cloud/

https://medium.com/@chamilad/load-balancing-and-reverse-proxying-for-kubernetes-services-f03dd0efe80

https://stackoverflow.com/questions/63470915/how-to-deploy-nginx-ingress-controller-to-kubernetes-master-nodes-by-using-cont

https://stackoverflow.com/questions/63470915/how-to-deploy-nginx-ingress-controller-to-kubernetes-master-nodes-by-using-cont

https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

Domain Name

deploy k8s dashboard

https://serverfault.com/questions/1031810/400-error-with-nginx-ingress-to-kubernetes-dashboard

https://serverfault.com/questions/1019919/how-to-properly-configure-access-to-kubernees-dashboard-behind-nginx-ingress

AWS config

https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/

• flannel troubleshooting

  • ports 8285 and 8472 ?

• ingress controller creation

  • arn configmap to bind IAM aws and RBAC k8s

• labels on aws for the cluster

• check that all needed ports for k8s / kubeadm are opened

• coredns containercreating

• hostNetwork true ? hostPort.enabled true ? kubeconfig path ? Kind DaemonSet ?

https://github.com/kubernetes/cloud-provider-aws/blob/master/docs/getting_started.md

Automatic provisionning (creation / deletion) of 3 ec2 instances

SECURITY

• Use git filter branch to delete sensitive informations

• AWS keys

• DockerHUB keys

• No passwords in clear

• DB is not accessible from the internet

Externalize config variables as much as possible