trainline / consul-deployment-agent Goto Github PK

As part of trainline/environment-manager#12, Environment Manager now supports the "disabling" of services - preventing any future infrastructure from installing particular services. This is done by adding a Status value to the key/value store.

To complete this feature the Consul Deployment Agent needs to first check for the existence of this property and ignore installations if the Status value is 'Disabled'.

Given that you have defined the following in healthchecks.yml:

 some-service-is-running:
    name: some-service-is-running
    local_script: is-running.ps1
    interval: 10

the check that will be generated by CDA will look like:

  "checks": {
    "order-notification-service-is-running": {
      "command": "C:\\temp\\e3c9cfb0-f92c-11e6-be86-8389fd3ece3c\\archive\\healthchecks\\sensu\\is-running.ps1",
      "interval": 10
    }
  }
}

when running this check, Sensu client will always timeout and eventually, by the look of it, stop. this is because Sensu client doesn't know how to run PowerShell scripts out of the box. looking at other system checks, e.g. consul-deployment-agent, we need to define the check as follows for it to run properly:

  "checks": {
    "order-notification-service-is-running": {
      "command": "powershell.exe -NonInteractive -NoProfile -ExecutionPolicy Bypass -file \"C:\\temp\\e3c9cfb0-f92c-11e6-be86-8389fd3ece3c\\archive\\healthchecks\\sensu\\is-running.ps1\"",
      "interval": 10
    }
  }
}

now, because of the way we have to define checks using the yaml file specification, i.e. local_script or server_script + script_arguments, it means that Windows users cannot include the powershell.exe -NonInteractive -NoProfile -ExecutionPolicy Bypass -file bit of the command as this won't match a filename on disk. this validation is done in CDA before registering a check.

We are getting alerts raised by Sensu shortly after deployments that are caused by how we are handling check scripts ans Sensu client's inability to pick up check definition changes quickly enough.

For example:

1. Deployment 1 defines a check with a script called my-check.sh
2. Deployment agent creates a Sensu check definition as such:

{
  "checks" : {
    "check-name": {
      "command": "/temp/deployments/1/healthchecks/sensu/my-check.sh"
    }
  }
}

3. Check definition is picked up by Sensu client
4. Later, deployment 2 redefines the same check with the same script
5. Deployment agent overwrites the Sensu check definition with:

{
  "checks" : {
    "check-name": {
      "command": "/temp/deployments/2/healthchecks/sensu/my-check.sh"
    }
  }
}

6. Previous directory /temp/deployments/1 is deleted by deployment agent
7. Sensu client takes a long time to pick up this change and results in the check failing due to the fact that /temp/deployments/1/healthchecks/sensu/my-check.sh no longer exists on disk.

Potential solutions:

1. Deployments should automate the silencing of checks until we know for sure that Sensu client has picked up the latest definitions. How we know that? I am not sure.
2. Deployment agent can more local check scripts to the deployment location of an application, e.g. /opt/application_name/healthchecks/sensu/my-script.sh. When redeploying, we would end up overwriting the existing script to ensure that the checks currently defined still work.

Hello,

It seems unzipping certain type or archives are not supported.

Here is the stracktrace hit while trying to deploy my app:

[2017-08-03 15:41:38,237] [DEBUG] Extracting /opt/consul-deployment-agent/deployments/dummy-service/dummy-id/bundle.zip to /opt/consul-deployment-agent/deployments/dummy-service/dummy-id/archive.
[2017-08-03 15:41:40,358] [ERROR] 'ascii' codec can't decode byte 0xf0 in position 19: ordinal not in range(128)
Traceback (most recent call last):
  File "agent/deployment_stages/common.py", line 19, in run
  File "agent/deployment_stages/download_bundle_from_s3.py", line 25, in _run
  File "zipfile.py", line 1024, in extract
  File "zipfile.py", line 1065, in _extract_member
  File "posixpath.py", line 80, in join
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 19: ordinal not in range(128)