My personal notes for Ops colleagues with tasks to do to dive into DevOps work.
- practical cases to learn new tools by doing;
- orienteers where to go, but you are free with how to go;
- but some of first tasks are more detailed to make the start easier.
Please, feel free to
- copy/fork/share/star this repository content;
- create issues / pull requests if you have any corrections, suggestions, ideas or if you want to discuss something.
Ensure you
- are self-education ready;
- don't afraid to make something work even in a simple/dirty way before making it better;
- know what's burnout at work (please, avoid this!), if you know Russian, see this and this videos from our IT colleagues about this;
- have Linux/Unix environment to work (any Linux, macOS, Windows with WSL or Linux VM onboard), here's my Linux WS setup example;
- have an IDE (I recommend VSCode with a lot of free plugins and Remote-SSH workspace feature);
- saw the DevOps Roadmap to know approximate path.
Checklist.
- I have personal GitHub.com account.
- I setup 2FA for my account.
- I generated and added to the account my SSH key for repository authentication (for pull & push).
- (optional) I generated and added to the account my GPG key for commit signing to have Verified badge for my commits to guarantee I'm an author of my commits (Git allows to set arbitrary
user.name
anduser.email
properties). - I set email settings to keep my email private in commits
Keep my email addresses private
Block command line pushes that expose my email
English language is very important for work. But English study is also a long way, so try to use it daily to improve your skills little by little.
- Reading: I use documentation in English as primary information sources.
- Writing: I write in English any of own docs, code and commit comments.
- Listening: I watch videos and courses by English speakers (at least with subtitles), e.g. on YouTube, Udemy.
- (recommended) Speaking: I have a English tutor or group for speaking practice at least for 1-2 hour 1-2 times in a week.
Create a simple web application using Flask web framework.
Key points and actions:
-
Create a new GitHub.com repository for this project.
-
Clone it using SSH.
-
Ensure you have
python3
available inPATH
. -
Create and commit requirements.txt file with pip package dependecies (
flask
). -
Create Python virtual environment (venv) in
./venv
directory, activate in the terminal and install requirements there -
Ensure
./venv
dir is in.gitignore
— such non-source data should never be committed. -
Write simplest Flask app which returns a string or HTML page on any (
/
) request. -
Run the app in the venv and check it is available on http://localhost:5000/.
-
Ensure you configured Git in your project dir
git config --local user.name "<GITHUB_USER_NAME>" # In (https://github.com/settings/emails use private email git config --local user.email "<GITHUB_USER_EMAIL>" # In case of GPG signature usage git config --local commit.gpgsign true git config --local user.signingkey <KEY_ID> # Env var for GPG key password prompt in a terminal session export GPG_TTY=$(tty)
-
Commit and push all the changes.
- Purchase an own domain (to reach your apps by name).
- Create an AWS account with 1 year free tier plan.
- Setup billing alerts to avoid unexpected surprises!
- Check what's included to free tier. E.g. don't launch big instances and don't run more that 1 instance for a long time if you don't want to exceed the limits!
- Go to EC2 service in web console.
- Choose a region for work.
- Create a security group with only
80
,443
and22
incoming ports opened (cloud firewall). - Create minimal EC2 instance with Linux: Ubuntu or Amazon Linux (rpm based distro).
- Create and assign Elastic IPv4 address to the instance (this is static external IP address).
- Create a DNS record for your domain pointing to the instance.
It's OK to do everything manually in a dirty way: make it work first.
- Connect to the instance via SSH using domain name as a host.
- Setup there your
PYTHON_APP
as you did it locally: git clone (by HTTPS wihtout auth if project is public), venv setup. - Run manually using built-in Flask web server (as you did it locally) with
0.0.0.0
host and80
port and enjoy your app available over global internet! Then stop it. - Run the project as systemd service using gunicorn or uWSGI web application server on
localhost:7000
. - Install NGINX and configure it to
- listen ports
80
and443
; - setup reverse proxy for all requests to application server on
localhost:7000
;
- listen ports
- Install
certbot
and get Let's Encrypt free TLS certificate. - Configure NGINX to
- use TLS certificate on
443
port; - redirect
80 --> 443
port;
- use TLS certificate on
- Enjoy your application available over HTTPS in internet!
- Sign up for Docker Hub — public images are free.
- Create a
Dockerfile
in the repository, it should- start from a Python 3 docker image;
- copy pip requirements file from the project into the image and install them;
- copy source code;
- run application web server as daemon on container start (read what the difference between
CMD
andENTRYPOINT
instructions);
- Install Docker locally. On macOS or Windows Docker Desktop is the simplest solution and it's still free for individual engineers, but there are a lot of
run docker without docker desktop
free solutions. - Build the Docker image locally.
- Run the image locally and map the port from a container to host to see the application in your web browser on
localhost:8080
. - Perform
docker login
to Docker Hub. - Tag and push the image to Docker Hub.
Be noted that Docker uses iptables firewall for own routing and you should be caution with firewall setup if you install and run Docker on hosts in internet. Here it should be OK because AWS provides cloud firewall rules.
- SSH to your host.
- Install Docker (no any problems on Linux).
- Run a container on
localhost:7001
(shoud differ from the port used by systemd app instance). - Reconfigure NGINX to serve the app from a Docker container.
On push to any branch excepting main
OR every pull request to main
branch run following CI actions
-
GitHub SuperLinter — universal linter for many types of files. It helps to find out a lot of interesting details.
-
Specific Python code linter (e.g. try
flake8
, configure it committing.flake8
file into the repository).Here also recommended to split requirements.txt into a group of files
requirements ├── base.txt # common packages ├── dev.txt # packages for development like linters that don't needed on production └── production.txt # packages for production including app web server
e.g.
# base.txt flask==1.1.2 # dev.txt -r base.txt flake8==3.8.4 mccabe==0.6.1 pycodestyle==2.6.0 pyflakes==2.2.0 # production.txt -r base.txt gunicorn==20.0.4
-
If linters are passed, push the image to Docker Hub.
- Docker Hub credentials must be passed via repository secrets!!!
- Put
latest
tag if the branch ismain
, ordev
tag otherwise.
- Terraform: create a new EC2 instance from code.
- Ansible: provision the instance after Terraform to make it run the application.
- Run Ansible from local WS firstly;
- Then make it as CD step in GitHub actions providing SSH access from a temporary GitHub Actions VM (via secrets), see example.
- CD: In a GitHub actions replace EC2 instance with Terraform then tune the host with Ansible example.
- To be continued...