GithubHelp home page GithubHelp logo

ministryofjustice / cotton Goto Github PK

View Code? Open in Web Editor NEW
3.0 32.0 2.0 221 KB

Project independent shared fabric extensions to bootstrap first VM and manage configuration within team

Python 38.81% Shell 61.19%

cotton's Introduction

cotton

Project independent shared fabric extensions to bootstrap first VM.

It solves three problems:
  • how to easily bootstrap VM in any supported environment
  • how to easily reach and manage this node
  • how to store - shared organisation config - shared project config - user unique/confidential config (typically used to store credentials only)

Depends on following fabric env variables:

env.provider_zone = 'aws_dev'
env.project = 'foo-dev'  # can also be a path
env.vm_name = 'foo.master'

#uncomment to always use shared provisioning key (only for early dev)
env.provisioning = True

Environment is provider configuration.

Assumes that your config directory is next to directory containing fabfile.py:

root/
|-- application-deployment/
|   `-- fabfile.py
|
|-- ~/.cotton.yaml / ${COTTON_CONFIG}
|-- config/projects/{env.project}/cotton.yaml
...
|-- config/projects/{env.project|split('/')[1]}/cotton.yaml
|-- config/projects/{env.project|split('/')[0]}/cotton.yaml
|-- config/projects/cotton.yaml
|-- config/cotton.yaml
|-- application-deployment/vagrant/cotton.yaml  # deprecated in favour to application-deployment/cotton.yaml
`-- application-deployment/cotton.yaml

I.e.:
env.project = nomis/pvb/production

cotton.yaml search path will look like:
root/
|
|-- ~/.cotton.yaml / ${COTTON_CONFIG}
|
|-- config/projects/nomis/pvb/production/cotton.yaml
|-- config/projects/nomis/pvb/cotton.yaml
|-- config/projects/nomis/cotton.yaml
|
|-- config/projects/cotton.yaml
|-- config/cotton.yaml
|
`-- application-deployment/cotton.yaml

example ~/.cotton.yaml:

provider_zones:
  aws_dev:
    driver: aws
    aws_access_key_id: 'TBV'
    aws_secret_access_key: 'TBD'
    ssh_key: /Users/aceventura/.ssh/default
  my_static_name:
    driver: static
    hosts:
      - name: master
        ip: 1.2.3.4
      - name: master-staging
        ip: 1.2.3.5
  aws_staging:
    image_id: ami-3a689f4d
    provisioning_ssh_key: ../config/default.pem
    provisioning_ssh_key_name: default
    provisioning_user: ubuntu
    gateway: 1.2.3.4
    instance_type: m1.small
    security_groups:
      - default
      - ssh
      - web-server
      - salt-master
    region_name: eu-west-1
    driver: aws

driver status

aws:fully implemented
static:fully implemented (a good fallback if api access is not available)
vcloud:only selection, status, filtering, termination, no provisioning part

cotton's People

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cotton's Issues

bootstrap_minion does not accept arguments (fab specifics)

fab staging workon:lb-01.staging1.pvb bootstrap_minion:10.1.21.11

Traceback (most recent call last):
File "/Users/mlmalawski/.virtualenvs/pvb-deploy/lib/python2.7/site-packages/fabric/main.py", line 743, in main
_args, *_kwargs
File "/Users/mlmalawski/.virtualenvs/pvb-deploy/lib/python2.7/site-packages/fabric/tasks.py", line 405, in execute
results[''] = task.run(_args, *_new_kwargs)
File "/Users/mlmalawski/.virtualenvs/pvb-deploy/lib/python2.7/site-packages/fabric/tasks.py", line 171, in run
return self.wrapped(_args, *_kwargs)
File "/Users/mlmalawski/repos/cotton/cotton/api.py", line 91, in inner
ret = func(_args, *_kwargs)
TypeError: bootstrap_minion() takes exactly 0 arguments (1 given)

salt shaker fails with: Formula marked as resolved but target XYZ didn't exist

having formula-requirements.txt:

[email protected]:ministryofjustice/admins-formula.git==v1.1.0
[email protected]:ministryofjustice/bootstrap-formula.git==v2.1.0
[email protected]:ministryofjustice/firewall-formula.git==v1.2.0-rc.pvb.1
[email protected]:ministryofjustice/salt-formula.git==v1.0.0
[email protected]:ministryofjustice/hosts-formula.git==v2.0.0
[email protected]:ministryofjustice/hardening-formula.git==v1.2.2
[email protected]:ministryofjustice/ntp-formula.git==v1.0.1
[email protected]:ministryofjustice/bootstrap-formula.git==v2.0.0
[email protected]:ministryofjustice/repos-formula.git==v1.1.1
[email protected]:ministryofjustice/jenkins-formula.git==v1.0.5
[email protected]:ministryofjustice/nginx-formula.git==v3.2.0
[email protected]:ministryofjustice/monitoring-formula.git==v1.0.3
[email protected]:ministryofjustice/sensu-formula.git==v4.4.2
[email protected]:ministryofjustice/utils-formula.git==v1.0.1
[email protected]:ministryofjustice/elasticsearch-formula.git==v1.3.1
[email protected]:ministryofjustice/redis-formula.git==v1.0.1

we get output:

(opg-zero)mateusz4moj-4:opg-basement-deply mlmalawski$ fab shaker
Fetching [email protected]:ministryofjustice/admins-formula.git done
Fetching [email protected]:ministryofjustice/bootstrap-formula.git done
Fetching [email protected]:ministryofjustice/firewall-formula.git done
Fetching [email protected]:ministryofjustice/salt-formula.git done
Fetching [email protected]:ministryofjustice/hosts-formula.git done
Fetching [email protected]:ministryofjustice/hardening-formula.git done
Fetching [email protected]:ministryofjustice/ntp-formula.git done
Fetching [email protected]:ministryofjustice/repos-formula.git done
Fetching [email protected]:ministryofjustice/jenkins-formula.git done
Fetching [email protected]:ministryofjustice/nginx-formula.git done
Fetching [email protected]:ministryofjustice/monitoring-formula.git done
Fetching [email protected]:ministryofjustice/sensu-formula.git done
Fetching [email protected]:ministryofjustice/utils-formula.git done
Fetching [email protected]:ministryofjustice/elasticsearch-formula.git done
Fetching [email protected]:ministryofjustice/redis-formula.git done
Fetching [email protected]:ministryofjustice/logstash-formula.git done
Fetching [email protected]:ministryofjustice/java-formula.git done
Fetching [email protected]:ministryofjustice/python-formula.git done
Fetching [email protected]:ministryofjustice/supervisor-formula.git done
Fetching [email protected]:ministryofjustice/apparmor-formula.git done
Traceback (most recent call last):
  File "/Users/mlmalawski/.virtualenvs/opg-zero/lib/python2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/Users/mlmalawski/.virtualenvs/opg-zero/lib/python2.7/site-packages/fabric/tasks.py", line 424, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/Users/mlmalawski/.virtualenvs/opg-zero/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/Users/mlmalawski/repos/cotton/cotton/salt_shaker.py", line 501, in shaker
    shaker_instance.install_requirements()
  File "/Users/mlmalawski/repos/cotton/cotton/salt_shaker.py", line 252, in install_requirements
    (repo_dir, _) = self.install_requirement(formula)
  File "/Users/mlmalawski/repos/cotton/cotton/salt_shaker.py", line 302, in install_requirement
    raise RuntimeError("%s: Formula marked as resolved but target '%s' didn't exist" % (formula['name'], target))
RuntimeError: python: Formula marked as resolved but target '/Users/mlmalawski/repos/opg-basement-deply/vendor/_root/python' didn't exist

Bootstrapping fails if 'provisioning' user has been deleted.

We should not be leaving the 'provisioning' user on our nodes, as it is a shared account.

However, when it is rightly deleted, this then prevents bootstrap_minion from working, as this depends on the newly created node only having 'provisioning' available for connection.

A workaround for this is to set the 'hosts.gw.ip' value in project.yaml to 'user@ip', where user is your regular admin user account.

To me, it would be better to always use your general user account except for key bootstrapping operations (like bootstrap_minion), where 'provisioning' is the correct account to use -- by convention.

bootstrap_master/bootstrap_minion - roles arg

Roles can not be used as fabric task argument.
In such case it's being intercepted by internal fabric roles mechanism.

PS: It would be interesting to leverage the fabric roles mechanism but it might not be possible.

silent shaker update

shaker is silently updating formulas versions when git repo is up to date but all is needed is to checkout a different baseline

See sense formula below:

(pvb-deploy)Mateuszs-MacBook-Pro:pvb-deploy mlmalawski$ fab check
[localhost] local: for d in vendor/formula-repos/*; do (export GIT_DIR=$d/.git; git fetch --tags -q 2>/dev/null; echo -n "$d: "; latest_tag=$(git describe --tags $(git rev-list --tags --max-count=1 2>/dev/null) 2>/dev/null || echo "no tags"); current=$(git describe --tags 2>/dev/null || echo "no tags"); echo "latest: $latest_tag; current: $current"); done
-n vendor/formula-repos/admins-formula: 
latest: v1.0.1; current: v1.0.1
-n vendor/formula-repos/bootstrap-formula: 
latest: v1.0.1; current: v1.0.1
-n vendor/formula-repos/elasticsearch-formula: 
latest: v1.0.2; current: v1.0.2
-n vendor/formula-repos/firewall-formula: 
latest: v1.1.0; current: v1.1.0
-n vendor/formula-repos/hardening-formula: 
latest: v1.0.3; current: v1.0.3
-n vendor/formula-repos/hosts-formula: 
latest: v2.0.0; current: v2.0.0
-n vendor/formula-repos/java-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/jenkins-formula: 
latest: v1.0.3; current: v1.0.3
-n vendor/formula-repos/logstash-formula: 
latest: v1.0.6; current: v1.0.6
-n vendor/formula-repos/metrics-formula: 
latest: v1.0.7; current: v1.0.7
-n vendor/formula-repos/monitoring-formula: 
latest: v1.0.1; current: v1.0.1
-n vendor/formula-repos/nginx-formula: 
latest: v1.1.0; current: v1.1.0
-n vendor/formula-repos/ntp-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/postfix-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/python-formula: 
latest: v1.0.2; current: v1.0.2
-n vendor/formula-repos/rabbitmq-formula: 
latest: v1.0.2; current: v1.0.2
-n vendor/formula-repos/redis-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/repos-formula: 
latest: v1.0.01; current: v1.0.01
-n vendor/formula-repos/ruby-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/sensu-formula: 
latest: v2.1.3; current: v2.1.2
-n vendor/formula-repos/sslloadbalancer-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/supervisor-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/utils-formula: 
latest: v1.0.0; current: v1.0.0

Done.
(pvb-deploy)Mateuszs-MacBook-Pro:pvb-deploy mlmalawski$ fab shaker

Done.
(pvb-deploy)Mateuszs-MacBook-Pro:pvb-deploy mlmalawski$ fab shaker

Done.
(pvb-deploy)Mateuszs-MacBook-Pro:pvb-deploy mlmalawski$ fab check
[localhost] local: for d in vendor/formula-repos/*; do (export GIT_DIR=$d/.git; git fetch --tags -q 2>/dev/null; echo -n "$d: "; latest_tag=$(git describe --tags $(git rev-list --tags --max-count=1 2>/dev/null) 2>/dev/null || echo "no tags"); current=$(git describe --tags 2>/dev/null || echo "no tags"); echo "latest: $latest_tag; current: $current"); done
-n vendor/formula-repos/admins-formula: 
latest: v1.0.1; current: v1.0.1
-n vendor/formula-repos/bootstrap-formula: 
latest: v1.0.1; current: v1.0.1
-n vendor/formula-repos/elasticsearch-formula: 
latest: v1.0.2; current: v1.0.2
-n vendor/formula-repos/firewall-formula: 
latest: v1.1.0; current: v1.1.0
-n vendor/formula-repos/hardening-formula: 
latest: v1.0.3; current: v1.0.3
-n vendor/formula-repos/hosts-formula: 
latest: v2.0.0; current: v2.0.0
-n vendor/formula-repos/java-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/jenkins-formula: 
latest: v1.0.3; current: v1.0.3
-n vendor/formula-repos/logstash-formula: 
latest: v1.0.6; current: v1.0.6
-n vendor/formula-repos/metrics-formula: 
latest: v1.0.7; current: v1.0.7
-n vendor/formula-repos/monitoring-formula: 
latest: v1.0.1; current: v1.0.1
-n vendor/formula-repos/nginx-formula: 
latest: v1.1.0; current: v1.1.0
-n vendor/formula-repos/ntp-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/postfix-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/python-formula: 
latest: v1.0.2; current: v1.0.2
-n vendor/formula-repos/rabbitmq-formula: 
latest: v1.0.2; current: v1.0.2
-n vendor/formula-repos/redis-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/repos-formula: 
latest: v1.0.01; current: v1.0.01
-n vendor/formula-repos/ruby-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/sensu-formula: 
latest: v2.1.3; current: v2.1.3
-n vendor/formula-repos/sslloadbalancer-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/supervisor-formula: 
latest: v1.0.0; current: v1.0.0
-n vendor/formula-repos/utils-formula: 
latest: v1.0.0; current: v1.0.0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.