itv / domed-city Goto Github PK

View Code? Open in Web Editor NEW

12.0 29.0 9.0 404 KB

Simple Terraform API wrapper in Ruby

License: Other

Ruby 100.00%

domed-city's People

Contributors

Stargazers

Watchers

Forkers

stefancocora bsnape xiii russell-io mhlias jocag jameswhinn farkasmate bertonlan

domed-city's Issues

Support TF 0.10.x

There are a few changes which we'll need to handle in dome since the cli args have changed somewhat:

Handle remote state changes
Implement native state locking

More will be added to this issue as they're encountered

dome should create plans directory

When you run dome for the first time it generally errors because you don't have a plans directory.

We could create the plans directory every where we have TF files but I think it would be better if dome just made sure the directory existed as part of the execution process (this would have the added advantage of allowing us to add plans to our gitignore)

Dome should fail with better error on missing TF binary

Currently if Terraform is not installed/not on the $PATH, Dome fails with misleading errors:

16:03:25 About to execute command: terraform get -update=true
16:03:25 something went wrong when pulling remote TF modules

simplify domed-city

at the moment we are wrapping the terraform subcommands in dome (dome -p => terraform plan)

I think if we only wrap terraform but not the subcommands, then we have more freedom of running specific terraform commands.
This will also help when they make breaking changes in next terraform versions, or they add new functionalities / flags / new commands.

Add travis support

To run the tests of course...

we should be forced to specify our version of TF

Apols if I mis-state the problem here.

As I understand it dome will work with whatever version of terraform is installed on your system. This is dangerous (we've seen lots of problems introduced by different people in the team accidentally using different versions in the past).

We should have somewhere (any itv.yaml at a higher directory?) where dome requires you to lock down your terraform version.

If the wrong version is installed it could then fail early and tell you to go upgrade.

SimpleDB locking mechanism should include $USER

In order to be able to better identify the state of a current lock,
as an operator,
I should be able to see the $USER which has implemented the lock,
and optionally use this $USER in the --unlock command

Add validation of current TF version

Previously we've seen issues where a newer version of Terraform was being used on a version of remote state that was created by an earlier version. We should potentially add something to dome which validates the currently used version of TF against a known good version, perhaps defined in a YAML file.

Support eyaml lookups

In the projects that predated the use of domed-city, we've used eyaml within Rake to lookup sensitive data, such as the Artifactory password, to pass to user-data during instance build/ LC updating. We should consider adding this functionality to domed-city, to ensure we're not committing sensitive data to our terraform config.

AWS credentials still not being unset?

@richbos was just having the issue where AWS keys in his env were being picked up instead of via ~/.aws/credentials

Notify if updates to your Terraform modules have been released

Lookup via itv.yaml.

Add manual option?

Automatic lookup on plan? Would this be annoying/slow if we do this every time?

Support `itv.yaml`

Allow the following to be defined in the itv.yaml

def team
  @account.split('-').first
end

def accounts
  %W(#{team}-dev #{team}-prd)
end

def non_production_environments
  %w(infradev dev sit qa qa1 qa2 stg)
end

def production_environments
  %w(infraprd prd)
end

The current environment and account will still need to be inferred from the current directory. They are currently created in the constructor

def initialize(directories = Dir.pwd.split('/'))
  @environment = directories[-1]
  @account     = directories[-2]
end

We will need to update the following methods to compare the calculated environment and account to those listed in itv.yaml

def valid_account?(account_name)
  ...
end

def valid_environment?(account_name, environment_name)
  ...
end

when extracting hiera values into the certs folder should error if not set

If a hiera value to be used as a cert isn't set dome just writes an empty file out to the certs directory.

This can be quite hard to spot. It would be better if dome errored in this situation.

Module depth should be overridable

We're currently hard coding the -module-depth to 1 .. which means if you have a set of modules that are going to make changes, you have restricted output, and it's difficult to tell what's going on:

Path: plans/my-amazing-plan.tf

~ module.users.aws_iam_group_membership.admins
    users.#:          "2" => "1"
    users.1234567890: "[email protected]" => ""
    users.1234567890: "[email protected]" => "[email protected]"

~ module.infra.private_subnet_nat
    2 resource(s)
~ module.sec_groups.sg_web
    1 resource(s)

Support assume-role

http://docs.aws.amazon.com/cli/latest/reference/sts/assume-role.html

Support locking

In the event of 2 users working in the same TF codebase, we modify the plan / apply methods with a a lock helper method that does:

def lock_tf_run
   raise("someone else is playing in that environment") unless there_is_no_lock_from_any_user_in_the_s3_bucket(bucket => "environment bucket")
   create_s3_object(filename => "username.lock",contents => "$timestamp", bucket => "environment bucket"
end

def unlock_tf_run
  delete_s3_object(filename => "username.lock", bucket => "environment bucket"
end

def create_plan
      command         = "terraform plan -module-depth=1 -refresh=true -out=#{@plan_file} -var-file=params/env.tfvars"
      failure_message = 'something went wrong when creating the TF plan'
      *lock_tf_run*
      execute_command(command, failure_message)
    end

def apply
      command         = "terraform apply #{@plan_file}"
      failure_message = 'something went wrong when applying the TF plan'
      execute_command(command, failure_message)
      *unlock_tf_run*
    end

What if there is no eyaml private key?

https://github.com/ITV/domed-city/blob/master/lib/dome/hiera_lookup.rb#L36-L44

Makes the assumption that you have an eyaml key and throws a stacktrace:

bundler: failed to load command: dome (/Users/timbirk/.gem/ruby/2.3.1/bin/dome)
NoMethodError: undefined method `colorize' for nil:NilClass
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/hiera_lookup.rb:42:in `eyaml_dir'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/hiera_lookup.rb:47:in `eyaml_private_key'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/hiera_lookup.rb:24:in `default_config'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/hiera_lookup.rb:13:in `config'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/hiera_lookup.rb:61:in `lookup'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/hiera_lookup.rb:73:in `block in secret_env_vars'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/hiera_lookup.rb:72:in `each_pair'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/hiera_lookup.rb:72:in `secret_env_vars'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/secrets.rb:15:in `secret_env_vars'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/terraform.rb:61:in `create_plan'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/lib/dome/terraform.rb:49:in `plan'
  /Users/timbirk/.gem/ruby/2.3.1/bundler/gems/domed-city-26657b98ba2f/bin/dome:30:in `<top (required)>'
  /Users/timbirk/.gem/ruby/2.3.1/bin/dome:23:in `load'
  /Users/timbirk/.gem/ruby/2.3.1/bin/dome:23:in `<top (required)>'

Which is not so useful.

Update readme with first-time usage instructions

e.g.

C02K50J5DRVG:stg devtoit$ dome -s
Account: cd-dev
Environment: stg
Team: cd
----------------------------------------------------------------
Enabling versioning on the S3 bucket - http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html
Putting an empty object with key: stg-terraform.tfstate into bucket: cd-tfstate-stg

Subsequent requests:

$ dome -s
Account: cd-dev
Environment: stg
Team: cd
----------------------------------------------------------------
Synchronising the remote S3 state...
About to execute command: terraform remote config -backend=S3 -backend-config='bucket=cd-tfstate-stg' -backend-config='key=stg-terraform.tfstate'
Remote configuration updated
Remote state configured and pulled.

Add changelog

Pass SIGINT gracefully to Terraform client

At the moment if I SIGINT domed-city midway through a terraform run, it won't propagate the signal to terraform.

Is this expected behaviour? Should I not expect to kill terraform via domed-city instead of letting its run its course?

module.vpc.aws_vpc_dhcp_options_association.env: Creation complete
module.vpc.aws_internet_gateway.igw: Creation complete
^C
~/working-dir/vendor/bundle/ruby/2.1.0/gems/domed-city-1.3.0/lib/dome/helpers/shell.rb:5:in `system'Interrupt received. Gracefully shutting down...
: Interrupt
    from ~/working-dir/vendor/bundle/ruby/2.1.0/gems/domed-city-1.3.0/lib/dome/helpers/shell.rb:5:in `execute_command'
    from ~/working-dir/vendor/bundle/ruby/2.1.0/gems/domed-city-1.3.0/lib/dome/terraform.rb:42:in `apply'
    from ~/working-dir/vendor/bundle/ruby/2.1.0/gems/domed-city-1.3.0/bin/dome:31:in `<top (required)>'
    from ~/working-dir/vendor/bundle/ruby/2.1.0/bin/dome:23:in `load'
    from ~/working-dir/vendor/bundle/ruby/2.1.0/bin/dome:23:in `<main>'

[user]$ module.route53_root_presentation.aws_route53_zone.root_zone: Creation complete

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: .terraform/terraform.tfstate

Error message is misleading when dome is executed in wrong directory

https://github.com/ITV/domed-city/blob/master/lib/dome/settings.rb#L6

raise('[*] itv.yaml does not exist') unless File.exist? itv_yaml_path

should be

raise("[*] #{itv_yaml_path} does not exist") unless File.exist? itv_yaml_path

or similar.

itv / domed-city Goto Github PK

domed-city's People

Contributors

Stargazers

Watchers

Forkers

domed-city's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs