emerleite / tsuru-bluegreen

A blue-green deployment plugin for tsuru client

tsuru-bluegreen

A blue-green deployment plugin for tsuru client

Dependencies

Python 2.7

Installation

tsuru plugin-install bluegreen https://raw.githubusercontent.com/emerleite/tsuru-bluegreen/1.4.4/src/bluegreen.py

Deployment methods

By default, the plugin deploys using git push. But if you define a deploy_dir key, inside the Application section of the configuration file, it uses tsuru app-deploy command instead.

Configuration

Tsuru bluegreen uses convention over configuration. It assumes your application backend is named using blue and green sufixes, as explained bellow:

Create a tsuru-bluegreen.ini in your application root with the following configuration:

[Application]
name: <your_app>
deploy_dir: <./build> <./build2>

[NewRelic]
api_key: <newrelic_api_key>
app_id: <newrelic_app_id>

[Grafana]
endpoint: <logstash_endpoint>
index: <logstash_index>

[WebHook]
endpoint: http://example.com
payload_extras: key1=value1&key2=value2

[Hooks]
before_pre: <command to run before 'pre' action>
after_pre: <command to run after a successful 'pre' action>
before_swap: <command to run before 'swap' action>
after_swap: <command to run after a successful 'swap' action>

[UnitsRemoval]
retry_times: 20 <how many times to retry removing a unit>
retry_sleep: 10 <how much time to wait between tries>

Note: if a NewRelic key's value is left blank, the plugin will try to get it from an environment variable (NEW_RELIC_API_KEY or NEW_RELIC_APP_ID).

'Application' section

Based on the name configuration value, you must have to have two tsuru applications and git remotes named: your_app**-blue** and your_app**-green**.

The deploy_dir configuration value is used with the --app-deploy flag. The default value is ..

'NewRelic' section

Notify New Relic about your deployment after swap. See NewRelic docs.

'Grafana' section

Notify Grafana about your deployment after swap. See Grafana docs.

'WebHook' section

POST to a WebHook after deployment swap. The payload is the defined payload_extras plus tag=<tag_value>.

'Hooks' section

Hooks are optional. They are ran before or after the corresponding actions, and everything sent to stdout and stderr is ignored. If a before hook fails (return value isn't zero), the action (pre/swap) is cancelled. If you want to run the pre/swap action independently of the before hook execution, you need to make sure it always returns 0.

Hooks must run inside a shell. If you want to run a curl command, for instance, you should do it inside a shell script:

$ cat script.sh
#! /bin/sh
curl http://example.com

$ cat tsuru-bluegreen.ini
[Application]
name: test

[Hooks]
before_pre: ./script.sh

$ tsuru bluegreen pre -t some-tag

In this case, if curl command fails, the pre action will be cancelled.

'UnitsRemoval' section

There's an issue when performing the swap of an app with multiple units. After the first unit is removed, errors are encountered when trying to remove the remaining processes.

According to tsuru contributors, this behavior is most likely do to the project's internal lock scheme.

This section defines how many times the plugin is going to retry removing a unit and how much time will it wait between tries. For example, to tell bluegreen to retry removing a unit twenty (20) times and to wait ten (10) seconds between each try, write this to you .ini fiel:

[UnitsRemoval]
retry_times: 20
retry_sleep: 10

Note: experimentation showed that small values for retry_sleep and large values for retry_times yields better usability.

Example

$ tsuru app-list

+---------------+-------------------------+---------------------------------------------------+--------+
| Application   | Units State Summary     | Address                                           | Ready? |
+---------------+-------------------------+---------------------------------------------------+--------+
| sample-blue   | 4 of 4 units in-service | sample.example.com, sample-blue.cloud.example.com | Yes    |
| sample-green  | 0 of 0 units in-service | sample-green.cloud.globoi.com                     | Yes    |
+---------------+-------------------------+---------------------------------------------------+--------+

$ git remote

sample-blue
sample-green

Usage

tsuru bluegreen --help

usage: tsuru bluegreen action [options]

Tsuru blue-green deployment (pre and live).

positional arguments:
  action                pre or swap

optional arguments:
  -h, --help            show this help message and exit
  -t [TAG], --tag [TAG] Tag to be deployed (default: master)

Tests

$ make testdeps
$ make test

Or, if you wish to use Docker;

$ make docker-test

tsuru-bluegreen's Issues

Check if app is locked before deploying

Before starting a new deploy, the plugin should check if the app is locked.

Allow send email when deploy

Send an email when deploy to blue/green and when swap

Handle errors on swap for applications with more than one CNAME

When calling 'swap' for an application with two or more CNAMEs, I want the rollback to add all CNAMEs back to the app in live state.

Ex.:
Before the swap:

• test-blue: 2 CNAMEs
• test-green: 0 CNAMEs

Output of swap:

Changing live application to test-green ...

Adding 9 'web' units to test-green ...
Error removing cname of test-blue. Aborting...

Removing 9 'web' units from test-green ...

After swap:

• test-blue: 1 CNAME
• test-green: 0 CNAME

Python 3 support

This travis test reports the problem: https://travis-ci.org/emerleite/tsuru-bluegreen/jobs/78406358

It should reduce the number of units of the "pre/blue"

Blue green is a very important feature but if you have an app with a large number of units (eg: 50+) you probably don't want to keep 100 units up costing money, so it would be great reduce the number of units of the "pre/blue" app after the swap

Add integrations tests to deploy_pre

Refactor to use a class and add some tests

Add integrations tests to deploy_swap

Set not live app to zero units

When blue or green app is not live, it should not have any running units.

Implement WebHook

[WebHook]
endpoint: http_endpoint
payload_extras: key=value&key=value

Units removal retries routine throws an error when setting 'retry_times = 0'

Description

When trying a swap operation, in case of a non-configured retry_times value (which implies retry_times = 0), the execution crashes showing the error:

File "/root/.tsuru/plugins/bluegreen", line 174, in remove_units_per_process_type
    Error removing '%s' units from %s in %d tries. Please, remove it manually.""" % (process_name, app, i)
UnboundLocalError: local variable 'i' referenced before assignment

Steps to reproduce

Try a swap operation with a missing [UnitsRemoval] .ini section or set retry_times = 0.

System settings

Python version: 2.7.17

Update New Relic deployment notification

New Relic deployment notification v1 was ended. The new API (v2) must be implemented.

https://rpm.newrelic.com/api/explore/application_deployments/create

NewRelic's deploy notification can only be configured with an API_KEY in plain text

Nowadays, in order to configure a deploy notification to NewRelic, an API_KEY must be stored in plain text in a project's tsuru-bluegreen.ini.

It doesn't seem to be a good practice... What if this could be done with environment variables?

I have a PR for implementing this, if you find it useful.

Compatibility with tsuru 0.7.0

It breaks, because TSURU_TARGET changed to use protocol://hostname, instead of hostname, and cname is now a list, instead of string.

Cancel action if before hook fails

If the before hook fails (return value != 0), the current action (pre or swap) shouldn't continue.

Pass environment variables to hooks

Script parameters, like tag, could be set as environment variables to hook commands.

Add support for apps with more than one process type

I want to support apps with more than one process type, i.e. more than one line at Procfile.

#Procfile
web: bundle exec puma -C config/puma.rb
resque: bundle exec rake resque:work QUEUE=*

Disable New Relic when Swap

Figure out a way to disable New Relic when Swap blue-green containers.

Use the updated tsuru swap feature

tsuru/tsuru#1372

Notify deploy to New Relic

Use the provided tag to notify a deploy to New Relic

Implement a way to give project general information

Something like tsuru bluegreen info

Which app is live, tag, url, etc.

Multiple environment support

If you have multiple environments, the blue-green should allow a pre deploy and live deploy to each environment.

Example:
sample
sample-staging

The command should allow:
tsuru bluegreen deploy pre -e staging

Error when your are not logged in

The script throws an error when we try to run it but not logged in...

Traceback (most recent call last):
  File "/Users/<username>/.tsuru/plugins/bluegreen", line 151, in <module>
    cnames = [bluegreen.get_cname(green), bluegreen.get_cname(blue)]
  File "/Users/<username>/.tsuru/plugins/bluegreen", line 24, in get_cname
    data = json.loads(response.read())
  File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
exit status 1