Before this, after pushing a commit to Gitlab, our developer need to login into AWS Opsworks console, create a deployment (in case of Rails application deployment, they need to write a custom JSON that define which branch to deploy, option to migrate database or not). I decide to use Gitlab CI to reduce the complicated works to make a deployment.
First of all, the architecture.
Generally, our architecture is divide into 3 major steps:
Step 1: Developer push a commit or make a merge request into Gitlab Step 2: The pipeline is contains 3 small step
- code review: Using code climate to evaluate source code quality (sorry but I'm still working on this step)
- Java maven build: In case of Java application, using mvn command to build java source code into a war file to deploy. Base on branch name, we will setup gitlab-ci to select appropriate profile to build. In the end, we will use aws-cli to push war file to s3 for deploying later
- Commence deployment: Using aws-cli to call Opsworks deploy command
Each step above will be executed inside a docker container Step 3: There is an external server that frequently check for deployment status for any update. When a deployment finish, it will notify that deployment status to developers via a Slack channel (we're using Slack at work)
First, enable pipelines for the project
In GitLab project panel, select Settings → Generals → Permissions→ Turn on [Pipelines] → Save changes
Then, create CI settings file (.gitlab-ci.yml) under project's root folder. Below is the sample file for rails application deployment.
stages:
- code-review
- test
- deploy
variables:
# You may put it in gitlab secrete variables
APP_NAME: "xxxxxxxx" #Staging application name from Opsworks console
#STG parameter
STG_STACK_ID: "xxxxxxxxx" #Staging stack ID from Opsworks console
STG_LAYER_ID: "xxxxxxxxxxxx" #Staging layer ID
STG_APP_ID : "xxxxxxxxx" #Staging application ID from Opsworks console
#PROD parameter
PROD_STACK_ID: "xxxxxxxxx" #Production stack ID from Opsworks console
PROD_LAYER_ID: "xxxxxxxxxxxx" #Production layer ID
PROD_APP_ID : "xxxxxxxxx" #Production application ID from Opsworks console
code review:
image: docker:latest
stage: code-review
variables:
DOCKER_DRIVER: overlay
services:
- docker:dind
script:
- docker info
- docker pull codeclimate/codeclimate
- docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 init
- docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 analyze -f json > codeclimate.json || true
artifacts:
paths: [codeclimate.json]
except:
- master
test_job:
stage: test
image: netmile/adgame-build:latest #This image includes maven and aws-cli
script:
- which aws
- ls -l ./
- pwd
except:
- master
deploy to staging with no migration:
stage: deploy
image: netmile/adgame-build:latest
environment: staging
script:
- aws opsworks --region us-east-1 create-deployment --stack-id $STACK_ID --app-id $APP_ID --layer-ids $LAYER_ID --command "{\"Name\":\"deploy\"}" --custom-json "{\"deploy\":{\"$APP_NAME\":{\"scm\":{\"revision\":\"$CI_COMMIT_REF_NAME\"}}}}" --comment "$CI_COMMIT_REF_NAME"
when: manual
only:
- /^f_*/ #future branch must have "f_" prefix
- dev
deploy to staging with migration:
stage: deploy
image: netmile/adgame-build:latest
environment: adgame_admin_staging
script:
- aws opsworks --region us-east-1 create-deployment --stack-id $STACK_ID --app-id $APP_ID --layer-ids $LAYER_ID --command "{\"Name\":\"deploy\"}" --custom-json "{\"deploy\":{\"$APP_NAME\":{\"scm\":{\"revision\":\"$CI_COMMIT_REF_NAME\"},\"migrate\":true}}}" --comment "$CI_COMMIT_REF_NAME"
when: manual
only:
- /^f_*/
- dev
deploy to production with no migration:
stage: deploy
image: netmile/adgame-build:latest
environment: adgame_admin_production
script:
- aws opsworks --region us-east-1 create-deployment --stack-id $STACK_ID --app-id $APP_ID --layer-ids $LAYER_ID --command "{\"Name\":\"deploy\"}" --custom-json "{\"deploy\":{\"$APP_NAME\":{\"scm\":{\"revision\":\"$CI_COMMIT_REF_NAME\"}}}}" --comment "$CI_COMMIT_REF_NAME"
when: manual
only:
- /^prod_*/ #release tag must have "prod_" prefix
deploy to production with migration:
stage: deploy
image: netmile/adgame-build:latest
environment: adgame_admin_production
script:
- aws opsworks --region us-east-1 create-deployment --stack-id $STACK_ID --app-id $APP_ID --layer-ids $LAYER_ID --command "{\"Name\":\"deploy\"}" --custom-json "{\"deploy\":{\"$APP_NAME\":{\"scm\":{\"revision\":\"$CI_COMMIT_REF_NAME\"},\"migrate\":true}}}" --comment "$CI_COMMIT_REF_NAME"
when: manual
only:
- /^prod_*/
In case of Java application:
stages:
- code-review
- test
- build and deploy
variables:
# You may put it in gitlab secrete variables
APP_NAME: "xxxxxxxx" #Staging application name from Opsworks console
#STG parameter
STG_S3_WAR_FOLDER: "s3://java-app/stg/" # use the same path with which you set in Opsworks console
STG_STACK_ID: "xxxxxxxxx" #Staging stack ID from Opsworks console
STG_LAYER_ID: "xxxxxxxxxxxx" #Staging layer ID
STG_APP_ID : "xxxxxxxxx" #Staging application ID from Opsworks console
#PROD parameter
PROD_S3_WAR_FOLDER: "s3://java-app/prod/"
PROD_STACK_ID: "xxxxxxxxx" #Production stack ID from Opsworks console
PROD_LAYER_ID: "xxxxxxxxxxxx" #Production layer ID
PROD_APP_ID : "xxxxxxxxx" #Production application ID from Opsworks console
code review:
image: docker:latest
stage: code-review
variables:
DOCKER_DRIVER: overlay
services:
- docker:dind
script:
- docker info
- docker pull codeclimate/codeclimate
- docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 init
- docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 analyze -f json > codeclimate.json || true
artifacts:
paths: [codeclimate.json]
except:
- master
test stage:
stage: test
image: netmile/adgame-build:latest
script:
- which aws
- ls -l ./
- pwd
except:
- master
# build and deployment stage
build and deploy to staging:
stage: build and deploy
environment: staging
image: netmile/adgame-build:latest
script:
- mvn -Dbranch=$CI_COMMIT_REF_NAME -P stg_properties clean compile war:war
- aws s3 cp ./target/adgms.war $STG_S3_WAR_FOLDER
- aws opsworks --region us-east-1 create-deployment --stack-id $STACK_ID --app-id $APP_ID --layer-ids $LAYER_ID --command "{\"Name\":\"deploy\"}" --comment "$CI_COMMIT_REF_NAME"
when: manual
only:
- /^f_*/
- dev
build and deploy to production:
stage: build and deploy
environment: production
image: netmile/adgame-build:latest
script:
- mvn -Dbranch=$CI_COMMIT_REF_NAME -P prod_properties clean compile war:war
- aws s3 cp ./target/adgms.war $PROD_S3_WAR_FOLDER
- aws opsworks --region us-east-1 create-deployment --stack-id $STACK_ID --app-id $APP_ID --layer-ids $LAYER_ID --command "{\"Name\":\"deploy\"}" --comment "$CI_COMMIT_REF_NAME"
when: manual
only:
- /^prod_.*/
Next, we will setup secrete variables for the project. What we need here is AWS access key and secrete key of a user that has permission to write into s3 bucket (for storing java war file) and permission to call Opsworks deploy command. The user's permission is defined below:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1434613487000",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::java-app",
"arn:aws:s3:::java-app/*"
]
},
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
},
{
"Sid": "Stmt1413336234000",
"Effect": "Allow",
"Action": "opsworks:CreateDeployment",
"Resource": [
*your layer ID's arn goes here*
]
}
]
}
After creating user with permission above, create a new access key and secrete key and set them up in Gitlab secrete variables like this
In CI/CD runners settings, take the runner registration token to use later in runner setup
In Gitlab runner server, run this command below
sudo gitlab-runner register -n --url http://<gitlab server URL> --registration-token [registration token goes here] --executor docker --description “docker-in-docker” --docker-image "docker:latest" --docker-volumes /var/run/docker.sock:/var/run/docker.sock
Now push a commit with a appropriate branch's name, go to CI/CD section and make your first deployment.
For the notification to Slack channel, we have 2 choice.
- We can use Lambda function with a CloudWatch event to trigger it like a normal crontab. This have a downside. The minimum setup in CloudWatch event is 60 seconds, so you can only run the job once in a minute. The bright side is, of course, you don't have to manage any server at all, just put the source code into Lambda and config it right and you good to go
- If you want to monitor opsworks deployment more precisely (for example every 10 seconds), you can launch a EC2 instance and config a crontab inside. (Just remember, if you call the aws api too many time in a period of time, you may get response exception, so choose the crontab frequent above 10 seconds)
I will go with option 1 because it easy to config and this notification part is not production critical.
In Lambda console, create a new function.
The function must have appropriate role. In securities credentials console, create a role with json file below.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"opsworks:Describe*"
],
"Resource": [
"*"
]
}
]
}
After creating Lambda function, copy the file OpsworkDeploymentCheck.js to the code IDE.
Then, we need to config function basic settings.
With 128MB of memory, it will take several seconds for the function to complete, so we need to change function timeout from 3 seconds (default) to 30 secs or something below 1 min.
In CloudWatch console, create an event rule that run every minute to trigger the Lambda function.
In targets configuration, choose the lambda function that we have created above.
That's all. Every time when a deployment occur in opsworks, there will be a message sent to slack channel.