The end-to-end-devops from mfkimbell

End-to-End DevOps Pipeline

Tools Used:

AWS Hosting of all CI/CD resources and permissions to those resources
EC2 Manage instances that run various servers for the application architecture
Maven Managing dependencies and compiling Java code
Jenkins Build and monitor automation of the application deployment process
Ansible Scripts for uploading new Docker Images and Kubernetes pulling of Images
Tomcat Manage server for Java application
Docker Containerize the application and server
Kubernetes Docker container management and fault tolerance with load balancer
EKS AWS management of Kubernetes
eksctl Command line management of EKS clusters on AWS CLI
kubectl Command line management of Kubernetes clusters

Purpose

Purpose was to make an end-to-end CICD pipeline for an application stored on Github through AWS. Changes are made to the application on Github. Jenkins polls every minute to see if changes occur, so, in under a minute, Jenkins will execute a job via Ansible to build and upload a new version of the Docker container onto DockerHub. After this is complete, another Jenkins CD job will use Ansible to tell the Kubernetes Bootstrap EC2 to pull the new image from DockerHub and deploy it on multiple pods (for high availability) and create a Load balancer users can access to use the application.

Here are all of the EC2 Instances running:

Here are all of the Jenkins Jobs working:

(For the working version of the app, only RegApp_CI_Job and RegApp_CD_Job are used)

Documentation from start to end:

As I will be deleting all of these AWS resources to prevent charges on my account, I have documented thoroughly this entire process.

SSH into an ec2 instance, and download jenkins and java (we create security groups that allow access from port 8080 for when we need to access our webapps

"service jenkins start" to start the jenkins server

access jenkins with public ip and port

Next, need to integrate github with jenkins

Install git on jenkins instance

Install github plugin on jenkins GUI

Roadblock: jenkins uses "master" not "main" by default

Confiure git on jenkins GUI

Java JDK on Maven from our EC2 server

Integrate maven with jenkins

Here are the three Jenkins Jobs I've created

Here we can see our build artifacts from the maven webapp job

Now we setup the Tomcat server similarly

Had to remove default permissions, and update it so we can have admin access. Also created some shortcuts for starting and stoppping the server

And then we login to the tomcat manager console

Then we set up jenkins with some secrets to access the tomcat server

Next we create a jenkins job "buildAndDeployServer" that builds the webapp on the tomcat server. After run, we see "webapp" now appears on the tomcat server.

And now we can see our webapp running on Tomcat:

But I ran this job MANUALLY. I want it to run every time I make changes to my code. So we set a build trigger for code changes in Jenkins. We can poll the software configuration management, which is a cron job that checks to see if the there are changes to the code, and only runs if there are changes. The current configuration I have set up uses a wildcard syntax to check the server every minute:

And here is a job running automatically after a code change:

I want to deploy Tomcat into a docker container

So I make a new EC2 instance and install docker on it.

We have the interal port as 8080, and the external port as 8081. So we need to update the security group of the docker-host.

Here we open ports 8081-9000:

I write a dockerfile to install java and tomcat on centos and then start the Tomcat server. This was a nightmare to do for some reason. Normally you would just pull a Official Docker Tomcat image, but there was a well-known issue with that apparently:

Changed my docker container to be accessed via password so I can connect to it via jenkins. SSH is best practice, but this was easier.

Now we need to build a Jenkins job to deploy artifacts on our docker container. Specifically, we are sending the .war webapp file over SSH:

Spent a couple hours experimenting here since I again ran into some issues:

Finally managed to get the webapp.war file on my docker instance

Now we create a new folder to copy the artifacts to in the EC2 instance:

We also update our Jenkins job to reflect that:

Now that the file is with our dockerfile, we can update it to send the .war file to the tomcat webapp folder.

Create a new container with these updates:

Now if we go to the URL, (publicIP):8086/webapp we can see our webapp:

However, I built this container manually, and I'd like jenkins to build it for me. So we set it to check for updates every minute, and remove and rebuild if there are changes to the code:

Now we are going to integrate Ansible as a deployment tool. We set up access from our Ansible EC2 to the Docker EC2 with it's private IP since we are in a VPC, so that's allowed:

We can see that Ansible was successfully able to acces the private IP:

Now we need to make it so Jenkins can integrate with Ansible. We need to make a new Jenkins job that this time excludes the "exec" jobs, since we will be handling the deployment on Ansible.

We give our Anisble EC2 it's own dockerfile with the same instructions as before on building the Tomcat server, build that image, and then build that container.

And the webapp sucessfully runs on the docker container in the Ansible EC2:

Now we need to create an Ansible Playbook to execute these commands automatically. We need to instruct Ansible to instruct the Docker container to pull a container from DockerHub. First off, we need to change the host IPs on the Ansible server so we can access a playbook file on the Ansible server, altering vi/ansible/hosts. We create groups based on where we want the execution to occur:

This is my basic playbook. I create a task to create an image and specify that if this playbook is available it can be executed in the docker directory.

After running the task, we can see the image has been added:

Now, we want to upload our image onto DockerHub. So I login to DockerHub on my Ansible EC2, and then manually push the image of interest.

And now we see it on DockerHub:

Now I'd like to incorporate these steps into Ansible Playbook. I create two new tasks on the playbook to tag and upload the image:

Now I alter my jenkins job to execute the Ansible Playbook, as well as adding a "Poll ACM: * * * * *" so it's run automatically with code changes:

Next, I create another playbook, deploy_regapp.yml, that graps the container from DockerHub.

And once again, the webapp is up and running, this time through Ansible Playbook.

Next, we need to update our playbook to remove current versions so that the container actually gets updated when there are changes.

And we add that playbook to the Jenkins job execs:

Finally, our webapp is up and running again, all done automatically through Jenkins and Ansible whenever a change is made.

Here are all of the Jenkins jobs so far, Copy_Artifacts_onto_Ansible is the recent one I was referring to:

Now, I want to incorporate Kubernetes for recovery/high-availability in case something happens to the Docker container. I make a new EC2 instance, and download awscli and eksctl.

Then I make an IAM role "eksctl" with EC2, Cloudformation, and IAM Full access, as well as administrator access for now.

I run "eksctl create cluster" along with other specifications, and we can see the cluster up and running in Cloudformation:

I run "kubectl run webapp --image=http" to create a webapp pod, and I run "kubectl create deployment demo-nginx --image=nginx --port=80 --replicas=2" to create an "nginx" deployment of two replica pods.

Then, I expose the application with a load balancer on port 80:

Through the external address, we can see the Nginx page:

It's better practice to use Manifest Files, so we'll implement that now. First we create a yml file with pod creation instructions:

As well as for a load balancer:

Now we want to make a deployment from the Docker container on DockerHub. Previously, we were using a known Docker Container (nginx), however, I want to use MY docker container.

Accessing our webapp from our new load balancer (produced by aws):

Now, because of the Rolling deployment, even if one pod goes down, a new pod will be deployed to take it's place:

Next, we want Ansible to run these commands instead of running them manually. First we want to create groups so Ansible can connect to the private IPs of the EC2 instances:

And now we can see Ansible is able to access all of these servers (by checking uptime):

I make an ansible playbook to execute on kubernetes server:

We also need a playbook for the service/loadbalancer:

Now that it's working, I want to execute these playbooks from Jenkins like before:

We can see before I had no pods up and running, and after running the job they are up and running again:

In hindsight, these should be merged, so now we only need to run kube_deploy.yml:

Now, on Jenkins, we need to update our (previously) deployment job and change it to instead use ansible to create and upload a new image onto DockerHub. The highlighted section is what will be deleted that was previously from the "Copy_Artifacts_Onto_Ansible" job: