vishu-25 / devops-project

A devops project made on AWS.

AWS-DevOps-Project

Source code for MAVEN web-app

https://github.com/vishu-25/hello-world-devops-src.git

Project Plan

Install Jenkins on EC2 instance

Prerequisites

1. EC2 Instance
   • With Internet Access
   • Security Group with Port 8080 open for internet
2. Java 11 should be installed

jenkins installation

Documentation

 sudo wget -O /etc/yum.repos.d/jenkins.repo \
 https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key
sudo yum upgrade
sudo yum install epel-release
sudo yum install java-11-openjdk
sudo yum install jenkins
sudo systemctl daemon-reload

Start Jenkins

# Start jenkins service
service jenkins start

# Setup Jenkins to start at boot,
chkconfig jenkins on

Accessing Jenkins

By default jenkins runs at port 8080, You can access jenkins at

http://YOUR-SERVER-PUBLIC-IP:8080

Configure Jenkins

• The default Username is admin
• Grab the default password
• Password Location:/var/lib/jenkins/secrets/initialAdminPassword
• Skip Plugin Installation; We can do it later
• Change admin password
  • Admin > Configure > Password
• Configure java path
  • Manage Jenkins > Global Tool Configuration > JDK
• Create another admin user id

Tomcat installation on EC2 instance

Pre-requisites

1. EC2 instance with Java 11

Install Apache Tomcat

1. Download tomcat packages from https://tomcat.apache.org/download-80.cgi onto /opt on EC2 instance

   Note: Make sure you change <version> with the tomcat version which you download.

   # Create tomcat directory
   cd /opt
   wget http://mirrors.fibergrid.in/apache/tomcat/tomcat-8/v8.5.35/bin/apache-tomcat-8.5.35.tar.gz
   tar -xvzf /opt/apache-tomcat-<version>.tar.gz

2. give executing permissions to startup.sh and shutdown.sh which are under bin.

   chmod +x /opt/apache-tomcat-<version>/bin/startup.sh 
   chmod +x /opt/apache-tomcat-<version>/bin/shutdown.sh

   Note: you may get below error while starting tomcat incase if you dont install Java
   Neither the JAVA_HOME nor the JRE_HOME environment variable is defined At least one of these environment variable is needed to run this program

3. create link files for tomcat startup.sh and shutdown.sh

   ln -s /opt/apache-tomcat-<version>/bin/startup.sh /usr/local/bin/tomcatup
   ln -s /opt/apache-tomcat-<version>/bin/shutdown.sh /usr/local/bin/tomcatdown
   tomcatup

Check point :

access tomcat application from browser on port 8080

• http://<Public_IP>:8080

Using unique ports for each application is a best practice in an environment. But tomcat and Jenkins runs on ports number 8080. Hence lets change tomcat port number to 8090. Change port number in conf/server.xml file under tomcat home

cd /opt/apache-tomcat-<version>/conf
# update port number in the "connecter port" field in server.xml
# restart tomcat after configuration update
tomcatdown
tomcatup

Check point :

Access tomcat application from browser on port 8090

• http://<Public_IP>:8090

1. Now application is accessible on port 8090. but tomcat application doesnt allow to login from browser. changing a default parameter in context.xml does address this issue

   #search for context.xml
   find / -name context.xml

2. Above command gives 3 context.xml files. comment () Value ClassName field on files which are under webapp directory. After that restart tomcat services to effect these changes. At the time of writing this lecture below 2 files are updated.

   /opt/tomcat/webapps/host-manager/META-INF/context.xml
   /opt/tomcat/webapps/manager/META-INF/context.xml
   
   # Restart tomcat services
   tomcatdown  
   tomcatup

3. Update users information in the tomcat-users.xml file goto tomcat home directory and Add below users to conf/tomcat-users.xml file

    <role rolename="manager-gui"/>
    <role rolename="manager-script"/>
    <role rolename="manager-jmx"/>
    <role rolename="manager-status"/>
    <user username="admin" password="admin" roles="manager-gui, manager-script, manager-jmx, manager-status"/>
    <user username="deployer" password="deployer" roles="manager-script"/>
    <user username="tomcat" password="s3cret" roles="manager-gui"/>

4. Restart serivce and try to login to tomcat application from the browser. This time it should be Successful

Docker Installation

Pre-requisites

1. Amazon Linux EC2 Instance

Installation Steps

1. Install docker and start docker services

   yum install docker -y
   docker --version 
   
   # start docker services
   service docker start
   service docker status

2. Create a user called dockeradmin

   useradd dockeradmin
   passwd dockeradmin

3. add a user to docker group to manage docker

   usermod -aG docker dockeradmin
   

Validation test

1. Create a tomcat docker container by pulling a docker image from the public docker registry

   docker run -d --name test-tomcat-server -p 8090:8080 tomcat:latest

Installing Docker on Amazon Linux server

Pre-requisites

1. Amazon Linux EC2 Instance

Installation Steps

1. Install docker and start docker services

   yum install docker -y
   docker --version 
   
   # start docker services
   service docker start
   service docker status

2. Create a user called dockeradmin

   useradd dockeradmin
   passwd dockeradmin

3. add a user to docker group to manage docker

   usermod -aG docker dockeradmin
   

Validation test

1. Create a tomcat docker container by pulling a docker image from the public docker registry

   docker run -d --name test-tomcat-server -p 8090:8080 tomcat:latest

Ansible Installation

Ansible is an open-source automation platform. It is very, very simple to set up and yet powerful. Ansible can help you with configuration management, application deployment, task automation.

Pre-requisites

1. An AWS EC2 instance (on Control node)

Installation steps:

on Amazon EC2 instance

1. Install python and python-pip

   yum install python
   yum install python-pip

2. Install ansible using pip check for version

   pip install ansible
   ansible --version

3. Create a user called ansadmin (on Control node and Managed host)

   useradd ansadmin
   passwd ansadmin

4. Below command grant sudo access to ansadmin user. But we strongly recommended using "visudo" command if you are aware vi or nano editor. (on Control node and Managed host)

5. Log in as a ansadmin user on master and generate ssh key (on Control node)

   sudo su - ansadmin
   ssh-keygen

6. Copy keys onto all ansible managed hosts (on Control node)

   ssh-copy-id ansadmin@<target-server>

7. Ansible server used to create images and store on docker registry. Hence install docker, start docker services and add ansadmin to the docker group.

   yum install docker
   
   # start docker services 
   service docker start
   service docker start 
   
   # add user to docker group 
   usermod -aG docker ansadmin
   

8. Create a directory /etc/ansible and create an inventory file called "hosts" add control node and managed hosts IP addresses to it.

Validation test

1. Run ansible command as ansadmin user it should be successful (Master)

   ansible all -m ping

Ansible integration with Jenkins

Prerequisites:

1. Ansible server
2. Jenkins Server

Part-01 Integration Setps

Install "publish Over SSH"

• Manage Jenkins > Manage Plugins > Available > Publish over SSH

Enable connection between Ansible and Jenkins

• Manage Jenkins > Configure System > Publish Over SSH > SSH Servers

  • SSH Servers:
    • Hostname:<ServerIP>
    • username: ansadm
    • password: *******

Test the connection "Test Connection" equisites:

1. Ansible server
2. Jenkins Server

Part-01 Integration Setps

Install "publish Over SSH"

• Manage Jenkins > Manage Plugins > Available > Publish over SSH

Enable connection between Ansible and Jenkins

• Manage Jenkins > Configure System > Publish Over SSH > SSH Servers

  • SSH Servers:
    • Hostname:<ServerIP>
    • username: ansadm
    • password: *******

Test the connection "Test Connection"

Configure Git pulgin on Jenkins

Git is one of the most popular tools for version control system. you can pull code from git repositories using jenkins if you use github plugin.

Prerequisites

1. Jenkins server

Install Git on Jenkins server

1. Install git packages on jenkins server

   yum install git -y

Setup Git on jenkins console

• Install git plugin without restart

  • Manage Jenkins > Jenkins Plugins > available > github

• Configure git path

  • Manage Jenkins > Global Tool Configuration > git

  Install & configure Maven build tool on Jenkins

Maven is a code build tool which used to convert your code to an artifact. this is a widely used plugin to build in continuous integration

Prerequisites

1. Jenkins server

Install Maven on Jenkins

1. Download maven packages https://maven.apache.org/download.cgi onto Jenkins server. In this case, I am using /opt/maven as my installation directory

• Link : https://maven.apache.org/download.cgi

   # Creating maven directory under /opt
   mkdir /opt/maven
   cd /opt/maven
   # downloading maven version 3.6.0
   wget http://mirrors.estointernet.in/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz
   tar -xvzf apache-maven-3.6.1-bin.tar.gz

1. Setup M2_HOME and M2 paths in .bash_profile of the user and add these to the path variable

   vi ~/.bash_profile
   M2_HOME=/opt/maven/apache-maven-3.6.1
   M2=$M2_HOME/bin
   PATH=<Existing_PATH>:$M2_HOME:$M2

Checkpoint

1. logoff and login to check maven version

   mvn --version

So far we have completed the installation of maven software to support maven plugin on the jenkins console. Let's jump onto Jenkins to complete the remaining steps.

Setup maven on Jenkins console

1. Install maven plugin without restart

• Manage Jenkins > Jenkins Plugins > available > Maven Invoker
• Manage Jenkins > Jenkins Plugins > available > Maven Integration

2. Configure maven path

• Manage Jenkins > Global Tool Configuration > Maven

Setup Kubernetes on Amazon EKS

You can follow same procedure in the official AWS document Getting started with Amazon EKS – eksctl

Pre-requisites:

• an EC2 Instance
• Install AWSCLI latest verison

1. Setup kubectl
   a. Download kubectl version 1.21
   b. Grant execution permissions to kubectl executable
   c. Move kubectl onto /usr/local/bin
   d. Test that your kubectl installation was successful

   curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
   chmod +x ./kubectl
   mv ./kubectl /usr/local/bin 
   kubectl version --short --client

2. Setup eksctl
   a. Download and extract the latest release
   b. Move the extracted binary to /usr/local/bin
   c. Test that your eksclt installation was successful

   curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
   sudo mv /tmp/eksctl /usr/local/bin
   eksctl version

3. Create an IAM Role and attache it to EC2 instance
   Note: create IAM user with programmatic access if your bootstrap system is outside of AWS
   IAM user should have access to
   IAM
   EC2
   CloudFormation
   Note: Check eksctl documentaiton for Minimum IAM policies

4. Create your cluster and nodes

   eksctl create cluster --name cluster-name  \
   --region region-name \
   --node-type instance-type \
   --nodes-min 2 \
   --nodes-max 2 \ 
   --zones <AZ-1>,<AZ-2>
   
   example:
   eksctl create cluster --name project-cluster \
      --region ap-south-1 \
   --node-type t2.small \

5. To delete the EKS clsuter

   eksctl delete cluster valaxy --region ap-south-1

6. Validate your cluster using by creating by checking nodes and by creating a pod

   kubectl get nodes
   kubectl run tomcat --image=tomcat 

   Deploying Nginx pods on Kubernetes

7. Deploying Nginx Container

   kubectl create deployment  demo-nginx --image=nginx --replicas=2 --port=80
   # kubectl deployment regapp --image=valaxy/regapp --replicas=2 --port=8080
   kubectl get all
   kubectl get pod

8. Expose the deployment as service. This will create an ELB in front of those 2 containers and allow us to publicly access them.

   kubectl expose deployment demo-nginx --port=80 --type=LoadBalancer
   # kubectl exp# Setup Kubernetes on Amazon EKS
   

You can follow same procedure in the official AWS document Getting started with Amazon EKS – eksctl

Pre-requisites:

• an EC2 Instance
• Install AWSCLI latest verison

1. Setup kubectl
   a. Download kubectl version 1.21
   b. Grant execution permissions to kubectl executable
   c. Move kubectl onto /usr/local/bin
   d. Test that your kubectl installation was successful

   curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
   chmod +x ./kubectl
   mv ./kubectl /usr/local/bin 
   kubectl version --short --client

2. Setup eksctl
   a. Download and extract the latest release
   b. Move the extracted binary to /usr/local/bin
   c. Test that your eksclt installation was successful

   curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
   sudo mv /tmp/eksctl /usr/local/bin
   eksctl version

3. Create an IAM Role and attache it to EC2 instance
   Note: create IAM user with programmatic access if your bootstrap system is outside of AWS
   IAM user should have access to
   IAM
   EC2
   CloudFormation
   Note: Check eksctl documentaiton for Minimum IAM policies

4. Create your cluster and nodes

   eksctl create cluster --name cluster-name  \
   --region region-name \
   --node-type instance-type \
   --nodes-min 2 \
   --nodes-max 2 \ 
   --zones <AZ-1>,<AZ-2>
   
   example:
   eksctl create cluster --name vismay25-cluster \
      --region ap-south-1 \
   --node-type t2.micro \

5. To delete the EKS clsuter

   eksctl delete cluster valaxy --region ap-south-1

6. Validate your cluster using by creating by checking nodes and by creating a pod

   kubectl get nodes
   kubectl run tomcat --image=tomcat 

   Deploying Nginx pods on Kubernetes

7. Deploying Nginx Container

   kubectl create deployment  demo-nginx --image=nginx --replicas=2 --port=80
   # kubectl deployment regapp --image=vismay25/regapp --replicas=2 --port=8080
   kubectl get all
   kubectl get pod

8. Expose the deployment as service. This will create an ELB in front of those 2 containers and allow us to publicly access them.

   kubectl expose deployment demo-nginx --port=80 --type=LoadBalancer
   # kubectl expose deployment regapp --port=8080 --type=LoadBalancer
   kubectl get services -o wide

   Integrating Kubernetes cluster with Ansible

9. Login to ansible server and copy public key onto kubernetes cluseter master account

10. Update hosts file with new group called kubernetes and add kubernetes master in that.

11. Create ansible playbooks to create deployment and services

12. Check for pods, deployments and services on kubernetes master

    kubectl get pods -o wide 
    kubectl get deploy -o wide
    kubectl get service -o wide

13. Access application suing service IP

    wget <kubernetes-Master-IP>:31200

Integration Kubernetes with Jenkins

Jenkins CI Job: Deploy_on_Kubernetes-CI

Pre-requisites

1. Jenkins server
2. Ansible server
3. Kubernetes cluster

Steps to create "Deploy_on_Kubernetes_CI" Jenkin job

From Jenkins home page select "New Item"

• Enter an item name: Deploy_on_Kubernetes_CI

  • Copy from: Deploy_on_Docker_Container_using_Ansible_playbooks

• Source Code Management:

  • Repository: https://github.com/yankils/hello-world.git
  • Branches to build : */master

• Poll SCM : - * * * *

• Build:

  • Root POM:pom.xml
  • Goals and options: clean install package

• Post-build Actions

  • Send build artifacts over SSH
    • SSH Publishers
    • SSH Server Name: ansible-server
    • Transfers > Transfer set
      • Source files: webapp/target/*.war
      • Remove prefix: webapp/target
      • Remote directory: //opt//docker
      • Exec command:

        ansible-playbook -i /opt/docker/hosts /opt/docker/create-simple-devops-image.yml --limit localhost;

Save and run the job.

Jenkins CD Job: Deploy_on_Kubernetes-CD

Steps to create "Deploy_on_Kubernetes_CI" Jenkin job

From Jenkins home page select "New Item"

• Enter an item name: Deploy_on_Kubernetes_CI

  • Freestyle Project

• Post-build Actions

  • Send build artifacts over SSH
    • SSH Publishers
      • Exec command:

        ansible-playbook -i /opt/docker/hosts /opt/docker/kubernetes-valaxy-deployment.yml;
        ansible-playbook -i /opt/docker/hosts /opt/docker/kubernetes-valaxy-service.yml;

Save and run the job.