[REQUEST] feature about runway

[ITProKyle]

I want to be able to have a central directory for my CFNgin configs instead of having them in the module ... attribute for config_directory

Have you tried place more than one config file in a module directory? All the config files are executed in alphabetical order which would be functionally the same as a central directory.

multiple modules that share much of the same configuration

have you tried using yaml anchors to use the shared configuration in more than one module?

deployments:
  - modules:
      - path: sampleapp-01.cfn
        parameters: &common-parameters
          common_value: value
          special_value: value
      - path: sampleapp-02.cfn
        parameters:
          <<: *common-parameters
          special_value: value2
    environments:
      dev: true
      prod: true
    regions
      - us-east-1

[animaxcg]

Yes we use multiple configs in a single module directory that isn't the use case. the use case is modules 1-N use many of the same configs and I would rather manage that in a central place instead of scattered in a dozen module directories and have duplicate entries.

Yaml anchors would work be would still be a pain to manage across a dozen modules but I would rather not have one big runway.yml file but have a runway.yml and dev.yml file for dev etc. keep stacker variables separate from run instruction file

If I have time I'll try to read through the code and create a PR for it and see what you think

[troyready]

@animaxcg Can you give an example (e.g. a contrived representation) of the shared configs for which you're looking?

Usually this sort of thing is better solved by putting the more general elements (i.e. the CloudFormation templates) in a shared place (e.g. elsewhere in the same repo, in another repo/archive, etc -- which Runway already supports), but we're definitely open to ideas here.

[animaxcg]

These aren't the cloudformation templates these are the stacker env files. in stacker itself you can pass in the config file to use.

in runway you cannot it is hard coded in runway/runway/cfngin.py where your stacker configs live. I can give a better example later

[animaxcg]

Submitted a PR probably needs much reviewing as I am not used to your standard. made sure it passed pylint one variable added to 2 classes, you may want to name changed. #187

[ITProKyle]

Thanks for the PR, it has provided additional insight. There appears to be a conflict in terminology being used both here in the issue and in the PR.

For clarification:

• CFNgin configuration files: The files that define what CFNgin does ( instruction files). They contain top-level keys such as namespace, sys_path, and stacks. They are the files where each stack that will be deployed is defined.
• environment files: Files containing variable values for a module. How or if these are used depends on the module type. These can empty or even nonexistent. They are one option used to control if a module is deployed for a specific environment/region. Their naming patten is $DEPLOY_ENVIRONMENT.env or $DEPLOY_ENVIRONMENT-$AWS_REGION.env for CFNgin.

From the PR, it appears you mean environment files instead of configuration files. For passing in shared variables, wither they are global, per-environment, per-region, or any combination of these, using parameters in the runway config file is going to be the best option. It can also be used in combination with the environment files which have limitations that parameters do not have (e.g. using any data type other than string or bool)

Parameters Example

This example would meet the requirements you outlined in the PR of a central place to store variable values and not having configuration in the runway config file. You would also be able to dynamically generate the runway.variables.yml file to dynamically fill it will values.

runway.yml

deployments:
  - modules:
      - path: sampleapp01.cfn
        environments:
          prod: false
      - path: sampleapp02.cfn
        environment:
          dev: ${var enable_sampleapp02.${env AWS_REGION}}
          prod: false
    parameters:
      namespace: example-${env DEPLOY_ENVIRONMENT}
      nested_variable: ${var ${env DEPLOY_ENVIRONMET}.${env AWS_REGION}.nested_map}
    regions:
      - us-east-1
      - us-west-2

runway.variables.yml

dev:
  us-east-1:
    nested_map:
      some_value: us-east-1-value
      a_list:
        - us-east-1-list
  us-west-2:
    nested_map:
      some_value: us-west-2-value
      a_list:
        - us-west-2-list

enable_sampleapp02:
  us-west-2: false

sampleapp01.cfn/config.yml

namespace: ${namespace}
cfngin_bucket: ''

sys_path: ./

stacks:
  some_stack:
    class_path: blueprints.app.BlueprintClass
    variables:
      MapValue: ${nested_variable}

---

If you don't want to use parameters, your next best option would be to store your variables in SSM parameter store for each account and hardcode their paths into lookups in the CFNgin config file. But, keep in mind that these lookups can only be used in specific places so they would need to be used in combination with environment files or parameters for things like top-level keys.

This wouldn't provide a central directory but would allow you to populate these values using an external system and SSM would act as a central data store to reference them.

SSM Example

sampleapp01.cfn/config.yml

namespace: ${namespace}
cfngin_bucket: ''

sys_path: ./

stack:
  some_stack:
    class_path: blueprints.app.BlueprintClass
    variables:
      MapValue: ${ssm /example/nested_variable::load=json, region=us-east-1}

---

Another option for a central directory would to be create a single directory for your environment files and symlink the required files into your module directories but this usage was made obsolete with the introduction of parameters and lookups in the runway config file.

[animaxcg]

Cool. figured out the sys_path while working on a custom hook yesterday. thanks for the feedback