GithubHelp home page GithubHelp logo

etcaterva / deployment Goto Github PK

View Code? Open in Web Editor NEW
3.0 3.0 3.0 40.39 MB

EtCaterva IaC scripts

Python 7.47% JavaScript 2.40% Shell 18.59% HCL 53.40% Jinja 18.14%
ansible playbook python etcaterva-servers terraform

deployment's Introduction

EtCaterva-Ansible

Set of Ansible recipes to setup and configure EtCaterva servers.

Why this repository?

Because having infrastructure to support physical backups is not cheap, and in the unlikely event of our server getting corrupted or compromised, we'll be able to set up a new one only executing one command.

To implement these 'setup scripts' we have choosen Ansible, due its flexibility, ease of use, and because some members of the team had already experience with it.

Now, we only have to care about losing data, having to maintain logical backups.

How to use it?

Using Ansible is pretty easy. We are aiming for the creation of Ansilbe playbooks to configure certain servers. These playbooks are located in the top level folder.

To install Ansible you can follow the Ansible install guide, or just try with a classic sudo apt-get install ansible (although I had problems with that version, having to install it using sudo pip install ansible).

To run a playbook run:

ansible-playbook -i <path/to/hosts.file> <path/to/playbook>

For example, to run the webservers.yml playbook (to configure the web servers) execute:

ansible-playbook -i hosts_dev webservers.yml

Note: For decrypting one of the variable files you will have to use either --ask-vault-pass or --vault-password-file. For more information see how to run a playbook with vault documentation.

If you need to pick the target username, you can simply add --extra-vars "ansible_user=root". For example, for an initial deployment of the production server, you can run:

ansible-playbook -i environments/prod/hosts full-site.yml --vault-password-file ../vault --extra-vars "ansible_user=root"

Testing changes

To make tesing the changes easier for us, we have created a script to automate the creation of a Virtual Machine, using Vagrant.

Note: Unfortunately, Ansible can't run on Windows, and given that we use the Ansible provisioner for Vagrant, you won't be able to use this Vagrantfile.

To install Vagrant (Debian):

sudo apt-get install vagrant

If you want to use Vagrant with libvirt (Debian):

# Install some dependencies needed
sudo apt-get install zlib1g-dev
sudo apt-get install nfs-kernel-server
sudo apt-get install libvirt-dev

# Install libvirt and mutate Vagrant plugins
vagrant plugin install vagrant-libvirt
vagrant plugin install vagrant-mutate

# Download the Vagrant box we are going to use, and mutate it for libvirt
vagrant box add ubuntu/trusty64
vagrant mutate ubuntu/trusty64 libvirt

Once you have Vagrant configured in your system, you will be able to run the scipt for creating the VM with:

vagrant up

The VM will get the IP 192.168.77.22. You can ssh using that IP and one of the created users, or using:

vagrant ssh

If you want to open the web servers, the following URLs (using xip.io) are configured to work:

deployment's People

Contributors

dnaranjo89 avatar mariocj89 avatar palvarez89 avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar

deployment's Issues

limit journalctl usage

$ realpath limits.conf 
/etc/systemd/journald.conf.d/limits.conf

$ cat limits.conf 
[Journal]
SystemMaxUse=500M
SystemKeepFree=100M
SystemMaxFileSize=100M
RuntimeMaxUse=100M
RuntimeKeepFree=100M
RuntimeMaxFileSize=200M

sudo systemctl restart systemd-journald

Arreglar google analytics

cloudflare esta haciendo esta cambiando el tracking de google analytics
de: UA-62791775-3
a : UA-62791775-2
el que acaba en 2 es el de EAS2 y el que acaba en 3 el de EAS3, y no se que esta haciendo cloudflare que se esta haciendo la picha un lio

este archivo: https://beta-dev.echaloasuerte.com/cdn-cgi/apps/head/Pzwi1A8fY0Sxix9Qft3XZy7kHHQ.js
eso lleva hardcoded ahi el tracking ID de la web antigua, no se de donde lo esta pillando... supongo que esta metido en algun sitio en cloudflare

Speed up Ansible on frontend CI

CI example: https://api.travis-ci.org/v3/job/546148668/log.txt

There a bunch of unnecessary steps here:

TASK [echaloasuerte-3 : Clone/update backend repository in /var/www/echaloasuerte-3/web-app] ***
ok: [54.38.215.150]
TASK [echaloasuerte-3 : Add xip.io to the ALLOWED_HOSTS list when TEST_DEPLOYMENT is defined] ***
skipping: [54.38.215.150]
TASK [echaloasuerte-3 : Manually create the initial virtualenv] ****************
ok: [54.38.215.150]
TASK [echaloasuerte-3 : Setup virtual environment for the app using requirements.txt] ***
ok: [54.38.215.150]
TASK [echaloasuerte-3 : Setup EXTRA deps needed] *******************************
ok: [54.38.215.150]
TASK [echaloasuerte-3 : Update /etc/hosts to add 'db' IP address] **************
ok: [54.38.215.150]
TASK [echaloasuerte-3 : Synchronize django app database] ***********************
ok: [54.38.215.150]
TASK [echaloasuerte-3 : Install docker python package] *************************
ok: [54.38.215.150]

and also some optimisations in docker could be done

The whole page is down in a eas3 frontend deployment

Every time there is a frontend deployment, the whole site is down for about 30s.

You can find here some logs of errors of this happening in some pages that are not eas3.
https://app.logz.io/#/goto/df7e6e92961a9ff5aa1d1c9b65e45d33?switchToAccountId=64805
image

@timestamp,"message","logzio_codec","log","level","@metadata","prospector","source","type","env","tags","input","beat","_logzio_insights"
--
"2019-07-02T06:41:18.242Z","""2019/07/02   08:41:17 [error] 9181#9181: *6477269 upstream prematurely closed connection   while reading response header from upstream, client: 172.68.94.6, server:   echaloasuerte.com, request: \""GET /favicon.ico   HTTP/1.1\"", upstream:   \""http://127.0.0.1:8081/favicon.ico\"", host:   \""echaloasuerte.com\"", referrer:   \""https://echaloasuerte.com/draw/new/number/\""""","""plain""","{""file"":{""path"":""/var/log/nginx/error.log""}}","""error""","{""beat"":""filebeat"",""type"":""doc"",""version"":""6.8.0""}","{""type"":""log""}","""/var/log/nginx/error.log""","""nginx""","""prod""","[""beats-5015"",""_grokparsefailure""]","{""type"":""log""}","{""name"":""prod-ovh2"",""hostname"":""prod-ovh2"",""version"":""6.8.0""}","[""8cc2e2523246d2f50d115c8e77285faff03b4563"",""0d419b4b56c9d64f1ee0610fe9b44ec076f0ea65"",""0e5634b3e3923f9c1f9fba9e1d2d28756ca4c36a""]"
"2019-07-02T06:41:11.238Z","""2019/07/02   08:41:09 [error] 9181#9181: *6477257 recv() failed (104: Connection reset by   peer) while reading response header from upstream, client: 172.68.94.6,   server: echaloasuerte.com, request: \""GET /favicon.ico   HTTP/1.1\"", upstream:   \""http://127.0.0.1:8081/favicon.ico\"", host:   \""echaloasuerte.com\"", referrer:   \""https://echaloasuerte.com/groups/\""""","""plain""","{""file"":{""path"":""/var/log/nginx/error.log""}}","""error""","{""beat"":""filebeat"",""type"":""doc"",""version"":""6.8.0""}","{""type"":""log""}","""/var/log/nginx/error.log""","""nginx""","""prod""","[""beats-5015"",""_grokparsefailure""]","{""type"":""log""}","{""name"":""prod-ovh2"",""hostname"":""prod-ovh2"",""version"":""6.8.0""}","[""0e5634b3e3923f9c1f9fba9e1d2d28756ca4c36a""]"
"2019-07-02T06:41:08.234Z","""2019/07/02   08:41:02 [error] 9181#9181: *6477237 recv() failed (104: Connection reset by   peer) while reading response header from upstream, client: 172.68.94.6,   server: echaloasuerte.com, request: \""GET /favicon.ico   HTTP/1.1\"", upstream:   \""http://127.0.0.1:8081/favicon.ico\"", host:   \""echaloasuerte.com\"", referrer:   \""https://echaloasuerte.com/draw/new/number/\""""","""plain""","{""file"":{""path"":""/var/log/nginx/error.log""}}","""error""","{""beat"":""filebeat"",""type"":""doc"",""version"":""6.8.0""}","{""type"":""log""}","""/var/log/nginx/error.log""","""nginx""","""prod""","[""beats-5015"",""_grokparsefailure""]","{""type"":""log""}","{""version"":""6.8.0"",""name"":""prod-ovh2"",""hostname"":""prod-ovh2""}","[""0e5634b3e3923f9c1f9fba9e1d2d28756ca4c36a""]"
"2019-07-02T06:40:53.226Z","""2019/07/02   08:40:52 [error] 9181#9181: *6477193 recv() failed (104: Connection reset by   peer) while reading response header from upstream, client: 172.68.94.6,   server: echaloasuerte.com, request: \""GET /favicon.ico   HTTP/1.1\"", upstream:   \""http://127.0.0.1:8081/favicon.ico\"", host:   \""echaloasuerte.com\"", referrer:   \""https://echaloasuerte.com/groups/\""""","""plain""","{""file"":{""path"":""/var/log/nginx/error.log""}}","""error""","{""beat"":""filebeat"",""type"":""doc"",""version"":""6.8.0""}","{""type"":""log""}","""/var/log/nginx/error.log""","""nginx""","""prod""","[""beats-5015"",""_grokparsefailure""]","{""type"":""log""}","{""name"":""prod-ovh2"",""hostname"":""prod-ovh2"",""version"":""6.8.0""}","[""0e5634b3e3923f9c1f9fba9e1d2d28756ca4c36a""]"
"2019-07-02T06:40:52.213Z","""2019/07/02   08:40:52 [error] 9181#9181: *6477191 recv() failed (104: Connection reset by   peer) while reading response header from upstream, client: 172.68.94.108,   server: echaloasuerte.com, request: \""GET /groups/   HTTP/1.1\"", upstream:   \""http://127.0.0.1:8081/groups/\"", host:   \""echaloasuerte.com\""""","""plain""","{""file"":{""path"":""/var/log/nginx/error.log""}}","""error""","{""beat"":""filebeat"",""type"":""doc"",""version"":""6.8.0""}","{""type"":""log""}","""/var/log/nginx/error.log""","""nginx""","""prod""","[""beats-5015"",""_grokparsefailure""]","{""type"":""log""}","{""version"":""6.8.0"",""name"":""prod-ovh2"",""hostname"":""prod-ovh2""}","[""0e5634b3e3923f9c1f9fba9e1d2d28756ca4c36a""]"
"2019-07-02T06:40:49.190Z","""2019/07/02   08:40:39 [error] 9181#9181: *6477152 recv() failed (104: Connection reset by   peer) while reading response header from upstream, client: 162.158.78.84,   server: echaloasuerte.com, request: \""GET   /static/media/groups_og_image.bda19441.png HTTP/1.1\"", upstream:   \""http://127.0.0.1:8081/static/media/groups_og_image.bda19441.png\"",   host: \""echaloasuerte.com\""""","""plain""","{""file"":{""path"":""/var/log/nginx/error.log""}}","""error""","{""beat"":""filebeat"",""type"":""doc"",""version"":""6.8.0""}","{""type"":""log""}","""/var/log/nginx/error.log""","""nginx""","""prod""","[""beats-5015"",""_grokparsefailure""]","{""type"":""log""}","{""name"":""prod-ovh2"",""hostname"":""prod-ovh2"",""version"":""6.8.0""}","[""0e5634b3e3923f9c1f9fba9e1d2d28756ca4c36a""]"

Use uwsgi --reload

This makes restarting uwsgi keep current connections

sudo uwsgi --reload /var/run/uwsgi1.pid

If we go for docker, then this will be irrelevant.

Initial deployment support for EAS 3.0

Current plan is to have an initial deployment of the new version of EAS so that new draws are handled by it, but legacy ones still go through the old frontend.

The integration idea would be:

  • docker for backend if feasible in current servers.
  • frontend compiled in static files at deployment time. As simple as possible but doing it inside docker would be handy and clean.
  • nginx to decide which frontend to use depending on the URL
  • nginx to configure new frontend as a single page app

In addition to this integration, it would be nice to have for development purposes a script to only deploy the new app (front + back)

Note: this doesn't have to be perfect, but has to be done soon.

Discard old Jenkins build.

Automatically remove old build from Jenkins ( there is an option, we just need. To add it to anaible scripts)

backups from prod to dev without github actions

Thanks to cryptojunkies cron jobs will get disabled if the project is not updated frequently.

Not looking forward to exploit github action to do what we need, we will try to run this in our servers.

Backup server

We host all our content in one single server. Data lost and outages are guaranteed. We should set up a backup server with a mongodb replica just in case.

uwsgi logs are misconfigured

At the moment the logs being generated by uwsgi in here and here are ever growing and dont have read permission for other.

We should probably rotate and purge them or find any other way to limit the size of them and change the permissions to allow any user to read them (or add all our users to a group that can read them).

At the moment, eas2 logs take 3GB in the disk.

Export database from prod to dev

Exporting the db will allow us to:

  • Gain confidence on the new code working wiht the prod data
  • allow us to check the data easier by enabling the admin mode in dev

Use staging settings in dev.echaloasuerte

Background

We use two environment variables to define the environment where the app will be running

NODE_ENV

Used to decide the type of server that we will be starting.
If NODE_ENV == production a production-like server is used. Technically it will be the same
if NODE_ENV != production a development server is used. This server is not meant to be used in production as it's watching the files for changes, doing hot reloading, etc

Possible values:

  • production
  • anything else

REACT_APP_ENV

Used to decide some settings such whether Google should index pages or which account the analytics events should be sent.

Possible values:

  • production (currently any deployed app, both in the prod and dev server)
  • local (running locally)
  • test (running battery tests)

What needs to be done

Right now we are always using the production settings in the deployed app (both in prod and dev).
We want to set that variable to REACT_APP_ENV=staging when deploying to dev.echaloasuerte.com

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.