GithubHelp home page GithubHelp logo

theonestack / cfhighlander Goto Github PK

View Code? Open in Web Editor NEW
25.0 6.0 24.0 392 KB

Cloudformation DSL and component library

License: MIT License

Ruby 97.05% HTML 1.93% Dockerfile 0.42% Makefile 0.60%
cloudformation dsl cfndsl ruby-gem automation aws aws-sdk modular cfhighlander cfndsl-templates

cfhighlander's People

Contributors

aaronwalker avatar cfln123 avatar dependabot[bot] avatar gitter-badger avatar guslington avatar mitchclarkebase2 avatar samseppiol avatar theonestackbot avatar toshke avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

toshke theonestackbot aaronwalker guslington aranga-nana conare rrakhmanto gitter-badger mcne65 vijesh12 lews rererecursive scott-o-brian jaredbrook mukesh23singh samseppiol cfln123 mitchclarkebase2 kyletbase2 cyberjames mjonesbase2 ahmadalwa koochinlim

cfhighlander's Issues

Support publishing additional artifacts

I would be useful to be able to include additional artifacts in the S3 publish for components.

Use Case: The api gateway component supports loading a api swagger file from S3 so it would be useful to just include the api swagger definition is the published templates instead of requiring an additional deployment step

maybe something like

PublishArtifact file: "my-swagger.yaml", key: "my-swagger-#{component_version}.yaml"

or

S3Artifact file: "my-swagger.yaml", key: "my-swagger-#{component_version}.yaml"

The key could be optional

find replace Ref with other function types such as FnImportValue when extending a template

What i would like to be able to do is extend a template and find and replace where a Ref is defined with the extended template and replace it with a FnImportValue.

Example cfhighlander template

CfhighlanderTemplate do
  Name 'service'

  Parameters do
    ComponentParam 'ServiceTag'
  end

  Extends 'ecs-service@latest' do
    replace name: 'EcsCluster', value: FnImportValue(FnSub("${EnvironmentName}-ECSCluster"))
    replace name: 'VPCId', value: FnImportValue(FnSub("${EnvironmentName}-VPC"))
    replace name: 'Listener', value: FnImportValue(FnSub("${EnvironmentName}-httpsListener"))
    parameter name: 'ServiceTag', value: Ref('ServiceTag')
  end

end

CfTemplateUrl can generate invalid cloudformation

When cfhighlander adds the CfTemplateUrl it can generate an invalid cloudformation output parameter name when a subcomponent stack is inlined. The name of the output variable needs to be correctly escaped when it get inlined

Support for building a Container Images and pushing to ECR for use with Lambda or ECS

This feature is kinda similar to LambdaFunctions in that it allows custom code to be package and deployed with the generated cloudformation but with support for building and publishing container images.

Use Case: Creating more application specific components and services. For example a component for deploying a running ProxySQL Rather than relying on public container images have the component include all the dependencies (Dockerfile, config etc)

Add a new operation to the highlander dsl to build and publish a container image

Something like

ContainerImage image_name: proxysql, dockerfile: 'Dockerfile', ecr_repo: '1234567890.dkr.ecr.eu-central-1.amazonaws.com'

During the compile stage highlander would run a docker build and docker push to ecr and would create the ecr repo if it didn't exist with options for setting the permissions for the ecr_repo

Support for stack dependson when inlining a component

Right now when we have component like

CfhighlanderTemplate do

  Parameters do
    ComponentParam 'RegionCode', 'eu', allowedValues: ['eu', 'us', 'ap']
  end

  Component template: 'a', name: 'a1', render: Inline do
    parameter name: 'RegionCode', value: Ref('RegionCode')
  end

  Component template: 'a', name: 'a2', dependson: ['a1'] do
    parameter name: 'RegionCode', value: Ref('RegionCode')
  end
    
end

We end up with an invalid DependsOn referencing the name of the inlined stack which has been removed.

Ideally it would be good to be able to reference a specific resource within the inlined stack maybe we could support for following syntax for dependson

  Component template: 'a', name: 'a2', dependson: ['a1.SnsTopic'] do
    parameter name: 'RegionCode', value: Ref('RegionCode')
  end

and the resulting DependsOn would just contain the reference to the inlined resource

@Guslington @toshke thoughts????

Rate exceeded build issue

Trying to deploy cloudformation update to development stack using jenkins, from a master branch. I can eventually get it to build after a few runs but this error does appear quite often.

/usr/local/bundle/gems/aws-sdk-core-3.21.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': Rate exceeded (Aws::CloudFormation::Errors::Throttling)
	from /usr/local/bundle/gems/aws-sdk-core-3.21.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
	from /usr/local/bundle/gems/aws-sdk-core-3.21.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
	from /usr/local/bundle/gems/aws-sdk-core-3.21.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
	from /usr/local/bundle/gems/aws-sdk-core-3.21.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
	from /usr/local/bundle/gems/aws-sdk-core-3.21.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
	from /usr/local/bundle/gems/aws-sdk-core-3.21.0/lib/seahorse/client/request.rb:70:in `send_request'
	from /usr/local/bundle/gems/aws-sdk-cloudformation-1.4.0/lib/aws-sdk-cloudformation/client.rb:3504:in `validate_template'
	from /usr/local/bundle/gems/highlander-0.1.2/lib/highlander.validator.rb:40:in `validate_local'
	from /usr/local/bundle/gems/highlander-0.1.2/lib/highlander.validator.rb:29:in `block in validate'
	from /usr/local/bundle/gems/highlander-0.1.2/lib/highlander.validator.rb:17:in `each'
	from /usr/local/bundle/gems/highlander-0.1.2/lib/highlander.validator.rb:17:in `validate'
	from /usr/local/bundle/gems/highlander-0.1.2/bin/highlander.rb:84:in `cfcompile'
	from /usr/local/bundle/gems/highlander-0.1.2/bin/highlander.rb:105:in `cfpublish'
	from /usr/local/bundle/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	from /usr/local/bundle/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	from /usr/local/bundle/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	from /usr/local/bundle/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	from /usr/local/bundle/gems/highlander-0.1.2/bin/highlander.rb:158:in `<top (required)>'
	from /usr/local/bundle/gems/highlander-0.1.2/bin/highlander:3:in `require_relative'
	from /usr/local/bundle/gems/highlander-0.1.2/bin/highlander:3:in `<top (required)>'
	from /usr/local/bundle/bin/highlander:23:in `load'
	from /usr/local/bundle/bin/highlander:23:in `<main>'

Support for using conditional on inlined Components

Right now conditional components only works for nested stacks but Ideally you would be able to define something as conditional and if it gets inlined the condition gets added to each resource that gets inlined.

Currently it correct adds the Parameter and Condition definitions in the inlined CloudFormation

Considering the following here component

CfhighlanderTemplate do

  Parameters do
    ComponentParam 'RegionCode', 'eu', allowedValues: ['eu', 'us', 'ap']
  end

  Component template: 'a', name: 'a1', render: Inline, conditional: true, enabled: false do
    parameter name: 'RegionCode', value: Ref('RegionCode')
  end

  Component template: 'a', name: 'a2', conditional: true, enabled: false do
    parameter name: 'RegionCode', value: Ref('RegionCode')
  end
    
end

and the cfndsl for component a

CloudFormation do

  Condition(:EURegion, FnEquals(Ref(:RegionCode), 'eu'))

  SNS_Topic(:SnsTopic) do
    DisplayName FnSub('topic-${RegionCode}')
  end

  SNS_Topic(:SnsTopic2) do
    Condition(:EURegion)
    DisplayName FnSub('topic2-${RegionCode}')
  end
end

The compiled template for b should be

---
AWSTemplateFormatVersion: '2010-09-09'
Conditions:
  Enablea1:
    Fn::Equals:
    - Ref: Enablea1
    - 'true'
  Enablea2:
    Fn::Equals:
    - Ref: Enablea2
    - 'true'
  EURegion:
    Fn::And:
    - Condtion: Enablea1
    - Fn::Equals:
      - Ref: RegionCode
      - eu
Resources:
  a2:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: "./a2.compiled.yaml"
      Parameters:
        RegionCode:
          Ref: RegionCode
    Condition: Enablea2
  SnsTopic:
    Type: AWS::SNS::Topic
    Properties:
      DisplayName:
        Fn::Sub: topic-${RegionCode}
    Condition: Enablea1
  SnsTopic2:
    Type: AWS::SNS::Topic
    Properties:
      DisplayName:
        Fn::Sub: topic2-${RegionCode}
    Condition: EURegion
Parameters:
  RegionCode:
    Type: String
    Default: eu
    NoEcho: false
    AllowedValues:
    - eu
    - us
    - ap
  Enablea1:
    Type: String
    Default: 'false'
    NoEcho: false
    AllowedValues:
    - 'true'
    - 'false'
  Enablea2:
    Type: String
    Default: 'false'
    NoEcho: false
    AllowedValues:
    - 'true'
    - 'false'
Description: b@latest - vlatest
Outputs:
  CfTemplateUrl:
    Value: "/b.compiled.yaml"
  CfTemplateVersion:
    Value: latest
  a1CfTemplateVersion:
    Value: latest

Parameter wiring refactor

Add support for outer component parameter wiring to allow for simpler components with no external component dependencies.

Support automatic output parameter wiring based output parameter discover removing the need to explicitly define a component parameter as an output parameter.

Also we should be able override this behaviour with something like

  Component template: 'ecs', name: 'myecscluster' do
    parameter name: 'SecurityGroupBastion' value: 'bation2.SGRandomName'
  end

using the same upstream components multiple times in a project

The issue is when you use the same component multiple times that is referencing an upstream component.

For example take a project that manages 2 s3-websites that includes the s3-cloudfront component twice.

CfhighlanderTemplate do
  Name 's3-website'
  Description "s3-website"

  Component name: 's3cloudfront1', template: 's3-cloudfront'
  Component name: 's3cloudfront2', template: 's3-cloudfront'

end

The s3-cloudfront component inlines the cloudfront component with the name hardcoded to cloudfront. https://github.com/theonestack/hl-component-s3-cloudfront/blob/master/s3-cloudfront.cfhighlander.rb#L11

There for the last cloudfront config will overwrite them all causing each s3-website to have the same configuration.

I think the way to resolve this would be to prefix the sub component names with the component name in the project. The same way we handled the config files

s3cloudfront1-cloudfront
s3cloudfront2-cloudfront

dns domain with the same name as a template causes highlander to create a parameter from an output

Issue

If my domain name is prefixed with the same same as one of my components, that value is turned into a Fn::GetAtt from the output of the component.

DnsDomain:
          Fn::GetAtt:
          - sftp
          - Outputs.mydomain

My work around for the moment is to apply that parameter at runtime. Do we need to allow for the functionality if we have the cfout() function to generate the Fn::GetAtt?

Example

CfhighlanderTemplate do
  Name 'my-sftp'
  Description "my-sftp"

  Component name: 'vpc', template: 'vpc' do
    parameter name: 'DnsDomain', value: "sftp.mydomain.com"
    parameter name: 'NetworkPrefix', value: '10'
    parameter name: 'StackMask', value: '16'
  end

  Component name: 's3', template: 's3'

  Component name: 'sftp', template: 'sftpDnsDomain' do
    parameter name: 'VpcId', value: cfout('vpc.VPCId')
    parameter name: 'SubnetIds', value: cfout('vpc.ComputeSubnets')
    parameter name: 'DnsDomain', value: "sftp.mydomain.com"
    parameter name: 'VPCCidr', value: cfout('vpc.VPCCidr')
  end

  Component name: 'loadbalancer', template: '[email protected]' do
    parameter name: 'VpcId', value: cfout('vpc.VPCId')
    parameter name: 'SubnetIds', value: cfout('vpc.PublicSubnets')
    parameter name: 'DnsDomain', value: "sftp.mydomain.com"
    parameter name: 'VpcEndpointIPs', value: cfout('sftp.VpcEndpointIPs')
  end

end

Exception

Validate template /Users/gus/src/ss/ss-sftp-cloudformation/out/yaml/ss-sftp.compiled.yaml locally
Traceback (most recent call last):
	21: from /Users/gus/.rbenv/versions/2.5.3/bin/cfcompile:23:in `<main>'
	20: from /Users/gus/.rbenv/versions/2.5.3/bin/cfcompile:23:in `load'
	19: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.9.0/bin/cfcompile:4:in `<top (required)>'
	18: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.9.0/bin/cfcompile:4:in `require_relative'
	17: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.9.0/bin/cfhighlander.rb:301:in `<top (required)>'
	16: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	15: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	14: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	13: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	12: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.9.0/bin/cfhighlander.rb:113:in `cfcompile'
	11: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.9.0/lib/cfhighlander.validator.rb:17:in `validate'
	10: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.9.0/lib/cfhighlander.validator.rb:17:in `each'
	 9: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.9.0/lib/cfhighlander.validator.rb:29:in `block in validate'
	 8: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.9.0/lib/cfhighlander.validator.rb:40:in `validate_local'
	 7: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/aws-sdk-cloudformation-1.19.0/lib/aws-sdk-cloudformation/client.rb:4203:in `validate_template'
	 6: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.51.0/lib/seahorse/client/request.rb:70:in `send_request'
	 5: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.51.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
	 4: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.51.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
	 3: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.51.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
	 2: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.51.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
	 1: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.51.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.51.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': Circular dependency between resources: [loadbalancer, vpc, sftp] (Aws::CloudFormation::Errors::ValidationError)

Configuration of inner components by filename

If there is componentname.config.yaml configuration file bystanding outer component template, this configuration should be automatically exported to componentname component, if such component exists, rather than having to indent such configuration under components / componentname / config key

Ability to provide raw cloudformation

Cfhighlander templates should not necessarily rely on cfndsl templates to produce cloudformation templates. Raw YAML and JSON templates should be allowed as well with little help of cfn2dsl

Test cases for components

Thinking about creating test cases with config to run in travis-ci builds. For example the sqs component has some default config to create some demo queues, where as that should be handled with a tests case defined in sqs.tests.yaml. Then maybe a seperate command cftest or cfcompile --config sqs.tests.yaml.

Skip validation of compiled inlined templates

When a sub component inlines another component all templates compiled are validated which can sometimes lead to validation errors of templates that are not required as they've been inlined.

for example the fargate-v2 component inlines the ecs-task component. If add a custom parameter to the fargate-v2 component that's used in the task definition, validation for the ecs-task template fails because it doesn't have the parameter.

However there is no point in validating the ecs-task template as it's been inlined into the fargate template.
I think we should delete all inlined templates before validation

Ruby constants autowiring not working

If two components, say A , and B have following defined

A - has an output defined as

# a.cfndsl.rb
Output(:InstanceId) { Value(Ref(:Instance)) }

B - has an input defined as

# b.cfhighlander.rb
Parameters { ComponentParam :InstanceId,'', type: 'String }

If C defines both A and B, output-input auto-wiring does not work. However if :InstanceId is replaced with 'InstanceId' this works.

cfhighlander issue when extending component with a mappings.yaml

cfhighlander version: 0.12.5

Project files

.
├── README.md
├── extend-error.cfhighlander.rb
├── extend-error.config.yaml
└── mysql
    ├── mysql.cfhighlander.rb
    ├── mysql.cfndsl.rb
    └── mysql.config.yaml

extend-error.cfhighlander.rb

CfhighlanderTemplate do
  Name 'extend-error'
  Description "extend-error"
	Component name: 'mysql', template: 'mysql'
end

mysql/mysql.cfhighlander.rb

CfhighlanderTemplate do
  Name 'mysql'
  Extends '[email protected]'
end

StackTrace

Config for mysql written to /Users/gus/src/testbed/hl/extend-error/out/config/mysql.config.yaml
cfndsl template for mysql written to /Users/gus/src/testbed/hl/extend-error/out/cfndsl/mysql.compiled.cfndsl.rb
Traceback (most recent call last):
	28: from /Users/gus/.rbenv/versions/2.7.2/bin/cfcompile:23:in `<main>'
	27: from /Users/gus/.rbenv/versions/2.7.2/bin/cfcompile:23:in `load'
	26: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/bin/cfcompile:4:in `<top (required)>'
	25: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/bin/cfcompile:4:in `require_relative'
	24: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/bin/cfhighlander.rb:305:in `<top (required)>'
	23: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	22: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	21: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	20: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	19: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/bin/cfhighlander.rb:96:in `cfcompile'
	18: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/bin/cfhighlander.rb:292:in `build_component'
	17: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.model.component.rb:156:in `load'
	16: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.model.component.rb:234:in `evaluateHiglanderTemplate'
	15: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.model.component.rb:234:in `eval'
	14: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.model.component.rb:248:in `evaluateHiglanderTemplate'
	13: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.dsl.template.rb:527:in `CfhighlanderTemplate'
	12: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.dsl.template.rb:238:in `loadComponents'
	11: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.dsl.template.rb:238:in `each'
	10: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.dsl.template.rb:269:in `block in loadComponents'
	 9: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.model.component.rb:255:in `eval_cfndsl'
	 8: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfhighlander-0.12.4/lib/cfhighlander.compiler.rb:141:in `evaluateCloudFormation'
	 7: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfndsl-1.3.1/lib/cfndsl/cloudformation.rb:58:in `eval_file_with_extras'
	 6: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfndsl-1.3.1/lib/cfndsl/cloudformation.rb:58:in `eval'
	 5: from /Users/gus/src/testbed/hl/extend-error/out/cfndsl/mysql.compiled.cfndsl.rb:30:in `eval_file_with_extras'
	 4: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfndsl-1.3.1/lib/cfndsl/cloudformation.rb:114:in `CloudFormation'
	 3: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfndsl-1.3.1/lib/cfndsl/jsonable.rb:164:in `declare'
	 2: from /Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfndsl-1.3.1/lib/cfndsl/jsonable.rb:164:in `instance_eval'
	 1: from /Users/gus/src/testbed/hl/extend-error/out/cfndsl/mysql.compiled.cfndsl.rb:57:in `block in eval_file_with_extras'
/Users/gus/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/cfndsl-1.3.1/lib/cfndsl/module.rb:79:in `block (4 levels) in dsl_content_object': wrong number of arguments (given 2, expected 1) as Mapping(EnvironmentType) already exists (ArgumentError)

Sub/Sub components failing to pickup config

This is my example

I have my base component which is cloud front
https://github.com/theonestack/hl-component-cloudfront

I then have another component that includes the cloudfront component and renders inline.
This is a s3 bucket and bucket policy
https://github.com/theonestack/hl-component-s3-cloudfront

i then have my project that specifies the s3-cloudfront component called s3-website

CfhighlanderTemplate do
  Name 's3-website'
  Description "s3-website"

  Component name: 's3cloudfront', template: 's3-cloudfront', render: Inline

end

I then want add config to the cloudfront component, so i create a cloudfront.config.yaml with my cloudfront config. However when i compile the config is not picked up. The compiled cloudfront config only contains config set by the cloudfront and s3-cloudfront components

I have tried specifying

component:
  cloudfront:

proving the config in a 's3-cloudfront.config.yaml' file

The only way i can get this to work is by vendoring the s3-cloudfront component in my s3-website project and proving the cloudfront config with that vendored component.

@toshke @aaronwalker any ideas why it might not be applying the config?

Create stack url

When doing cfhighlander cfpublish template[@version] command output should contain url for launching stack within same region where distribution bucket is, allowing an easy flow from cli to aws console

Improve error messaging

In case of cfndsl compilation failure, it may be worthwhile adding begin / rescue block, with informative message that possible case of compilation failure are outdated cfndsl resources defintions, and suggestion to user that this can be resolved by executing cfndsl -u

Component inheritance

Cfhighlander should support building components as extensions of other components. Config, highlander template and cloudformation template are inherited from parent component. Child component should be able to override config and parameters. Cfndsl template of child component should be appended to the end of cfndsl template belonging to inherited component. This will results in ability to override cloudformation resources in child template by giving them same resource names.

Feature request: pass Conditions through to templates

I have a use case where I need to configure the components for CloudFront and ACM based on their environment type. This can be achieved for each component individually by vendoring tem and then placing a Condition statement inside their templates. However, this is not a dynamic approach as it requires hard-coding the condition.

What would be nice is to have a dynamic Condition that is passed down to the desired templates. This can then be referenced in a component's config.

An example of this may look like:

is_prod = Condition('IsProd', FnEquals(Ref('EnvironmentType'), 'production'))

Component name: 'acm', template: 'acm' do
  parameter name: 'DnsDomain', value: 'hello.com'
  condition name: 'IsProd', value: is_prod
end

Error on compile due to new az_create_subnets method

A change to the subnet helper method seems to cause intermittent issues with components requiring an earlier version of VPC. See stack trace below.

Even though we are specifying component [email protected], bastion component has a dependson [email protected]. The addition of tags to the helper methods seems to throw an error.

      Component name:'vpc', template: '[email protected]' do
        parameter name: 'DnsDomain', value: dns_domain
      end

      Component name:'bastion', template: 'bastion' do
        parameter name: 'DnsDomain', value: dns_domain
        parameter name: 'InstanceType', value: 't2.micro'
      end

VPC helper method changes: theonestack/hl-component-vpc#18

Traceback (most recent call last):
        24: from /Users/jh/.rbenv/versions/2.5.3/bin/cfcompile:23:in `<main>'
        23: from /Users/jh/.rbenv/versions/2.5.3/bin/cfcompile:23:in `load'
        22: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/bin/cfcompile:4:in `<top (required)>'
        21: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/bin/cfcompile:4:in `require_relative'
        20: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/bin/cfhighlander.rb:298:in `<top (required)>'
        19: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
        18: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
        17: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
        16: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
        15: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/bin/cfhighlander.rb:107:in `cfcompile'
        14: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/lib/cfhighlander.compiler.rb:151:in `compileCloudFormation'
        13: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/lib/cfhighlander.compiler.rb:151:in `each'
        12: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/lib/cfhighlander.compiler.rb:152:in `block in compileCloudFormation'
        11: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/lib/cfhighlander.compiler.rb:147:in `compileCloudFormation'
        10: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.8.2/lib/cfhighlander.compiler.rb:123:in `evaluateCloudFormation'
         9: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfndsl-0.16.12/lib/cfndsl.rb:87:in `eval_file_with_extras'
         8: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfndsl-0.16.12/lib/cfndsl.rb:87:in `eval'
         7: from /Users/jh/clients/gcs-cloudformation/out/cfndsl/vpc.compiled.cfndsl.rb:14:in `eval_file_with_extras'
         6: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfndsl-0.16.12/lib/cfndsl.rb:93:in `CloudFormation'
         5: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfndsl-0.16.12/lib/cfndsl/jsonable.rb:204:in `declare'
         4: from /Users/jh/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfndsl-0.16.12/lib/cfndsl/jsonable.rb:204:in `instance_eval'
         3: from /Users/jh/clients/gcs-cloudformation/out/cfndsl/vpc.compiled.cfndsl.rb:179:in `block in eval_file_with_extras'
         2: from /Users/jh/clients/gcs-cloudformation/out/cfndsl/vpc.compiled.cfndsl.rb:179:in `each'
         1: from /Users/jh/clients/gcs-cloudformation/out/cfndsl/vpc.compiled.cfndsl.rb:184:in `block (2 levels) in eval_file_with_extras'
/Users/jh/.cfhighlander/components/vpc/1.2.0/ext/cfndsl/az.rb:104:in `az_create_subnets': wrong number of arguments (given 6, expected 2..5) (ArgumentError) ```

component_name not set when directly rendering a component

If I try and use component_name inside a highlander component and when rendering the component directly (not nested) I get the following error

$ ../cfhighlander/bin/cfhighlander cfcompile ecs-services --validate
Trying to load ecs-services@latest from /Users/aaronwalker/Workspaces/theonestack/hl-component-ecs-services/ecs-services ... 
Trying to load ecs-services@latest from /Users/aaronwalker/Workspaces/theonestack/hl-component-ecs-services ... 
Processing higlander component ecs-services
	Location:/Users/aaronwalker/Workspaces/theonestack/hl-component-ecs-services/ecs-services.highlander.rb
	Config:{"log_retention"=>7, "cpu"=>1024, "memory"=>256, "task_definition"=>{"nginx"=>{"image"=>"nginx", "ports"=>[80]}}, "targetgroup"=>{"name"=>"DefaultHTTP", "container"=>"nginx", "port"=>80}}
/Users/aaronwalker/Workspaces/theonestack/cfhighlander/lib/highlander.dsl.base.rb:18:in `method_missing': #<Highlander::Dsl::Template:0x007fbb72de12c0>Unknown method or variable component_name in Highlander template (StandardError)
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/lib/highlander.factory.rb:106:in `block in load'
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/lib/highlander.dsl.rb:419:in `instance_eval'
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/lib/highlander.dsl.rb:419:in `HighlanderComponent'
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/lib/highlander.factory.rb:102:in `load'
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/lib/highlander.factory.rb:102:in `eval'
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/lib/highlander.factory.rb:102:in `load'
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/bin/highlander.rb:154:in `build_component'
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/bin/highlander.rb:75:in `cfcompile'
	from /Users/aaronwalker/.chefdk/gem/ruby/2.4.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	from /Users/aaronwalker/.chefdk/gem/ruby/2.4.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	from /Users/aaronwalker/.chefdk/gem/ruby/2.4.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	from /Users/aaronwalker/.chefdk/gem/ruby/2.4.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	from /Users/aaronwalker/Workspaces/theonestack/cfhighlander/bin/highlander.rb:158:in `<top (required)>'
	from ../cfhighlander/bin/cfhighlander:3:in `require_relative'
	from ../cfhighlander/bin/cfhighlander:3:in `<main>'

Miss leading error if a parameter reference cannot be found

stack trace

Traceback (most recent call last):
	18: from /Users/gus/.rbenv/versions/2.5.3/bin/cfcompile:23:in `<main>'
	17: from /Users/gus/.rbenv/versions/2.5.3/bin/cfcompile:23:in `load'
	16: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/bin/cfcompile:4:in `<top (required)>'
	15: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/bin/cfcompile:4:in `require_relative'
	14: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/bin/cfhighlander.rb:301:in `<top (required)>'
	13: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	12: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	11: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	10: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	 9: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/bin/cfhighlander.rb:110:in `cfcompile'
	 8: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/cfhighlander.compiler.rb:163:in `compileCloudFormation'
	 7: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/util/cloudformation.util.rb:26:in `flattenCloudformation'
	 6: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/util/cloudformation.util.rb:280:in `collect_replacements'
	 5: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/util/cloudformation.util.rb:280:in `each'
	 4: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/util/cloudformation.util.rb:285:in `block in collect_replacements'
	 3: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/util/cloudformation.util.rb:285:in `each'
	 2: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/util/cloudformation.util.rb:290:in `block (2 levels) in collect_replacements'
	 1: from /Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/util/cloudformation.util.rb:290:in `each'
/Users/gus/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/cfhighlander-0.10.1/lib/util/cloudformation.util.rb:293:in `block (3 levels) in collect_replacements': undefined method `inlined' for nil:NilClass (NoMethodError)

Actual issue

cfhighlander.rb

  Component template: 'vpc', name: 'vpc' do
    parameter name: 'NetworkPrefix', value: '10'
    parameter name: 'StackMask', value: '16'
    parameter name: 'DnsDomain', value: dns_domain
    parameter name: 'dnszoneAddNSRecords', value: 'true'
  end

  Component template: 'ecs', name: 'ecs' do
    parameter name: 'SubnetIds', value: cfout('vpc.ComputeSubnets')
    parameter name: 'AsgMin', value: '1'
    parameter name: 'AsgMax', value: '2'
  end

  Component template: 'aurora-postgrest', name: 'postgres' do
    parameter name: 'DnsDomain', value: dns_domain
    parameter name: 'SubnetIds', value: cfout('vpc.PersistenceSubnets')
    parameter name: 'EnableReader', value: 'false'
    parameter name: 'ReaderInstanceType', value: ''
    parameter name: 'SecurityGroupBastion', value: cfout('bastion.SecurityGroupBastion')
    parameter name: 'SecurityGroupECS', value: cfout('ecs.EcsSecurityGroup')
  end

problem line
parameter name: 'SecurityGroupBastion', value: cfout('bastion.SecurityGroupBastion')
where bastion component does not exist.

Compilation errors

Opening this ticket so users can report failed compilations. This should allow us to improve error messaging system - #39

cfndsl 0.17.3 and above breaks

cfndsl/cfndsl#420 has changed the way the config is evaled into the template which no longer works for cfhighlander.

cfndsl/cfndsl#422 didn't quite solve the issue so the alternative suggested is to move to using the external_parameters config and disable the binding.

I have locked to 0.17.2 #120 for the time being, but we should do some testing with the pre 1.0 release

when inlining a component template the nested stack also gets included when the name of the component has a underscore or hypenen in the name

For example

CfhighlanderTemplate do
     Component template: 'c2', name: 'c3-test',  render: Inline
end

renders

---
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  c3test:
    Properties:
      TemplateURL: "./c3-test.compiled.yaml"
      Parameters:
        c1OutParam:
          Ref: c3testc1OutParam
    Condition: Enablec3test
    Type: AWS::CloudFormation::Stack
  Bucket:
    Properties:
      BucketName:
        Fn::Join:
        - ''
        - - Ref: c3testc1OutParam
          - "-c2"
    Type: AWS::S3::Bucket
Parameters:
  Enablec3test:
    Type: String
    Default: 'false'
    NoEcho: false
    AllowedValues:
    - 'true'
    - 'false'
  c3testc1OutParam:
    Type: String
    Default: ''
    NoEcho: false
Description: c4@latest - vlatest
Conditions:
  Enablec3test:
    Fn::Equals:
    - Ref: Enablec3test
    - 'true'

Tagging resources

I think there needs to be functionality in highlander core to manage tagging on an individual resource level through configuration.

My example is say the ecs template has a Name tag of Name: ${Environment}-ecs-xx but i want that to be Name: ${Environment}-${cluster}-ecs-xx. We could build that functionality into the template but i feel it would be something we'd repeat in a lot of templates.

Some example config for setting tags

Tags:
  Project: myApp # Tag is added to all resources that can have tags
  EcsAsg: # Resource Name
    Name: 
      Sub: ${Environment}-${Cluster}-ecs-xx

Also the ability to set global tags across all resources that could contain tags would be a good thing to have as well. However that maybe a difficult task as not all cloudformation resources use the Tag property to declare tags

Exclude compiled components used to inline from being published 

CfhighlanderTemplate do
  Name 'myProject'

  Component template: 'vpc', name: 'vpc', render: Inline

end

the following project produces 2 compiled artefacts and when cfpublish is run, both compiled templates are published to the s3 buckets. It would be nice to exclude the templates not referenced as a sub stack.

Component inlining

Having two inner components in outer component should optionally result in producing flattened cloudformation, composited from cloudformation templates of inner components. Complexity around this task is actual input/output wiring as cloudformation model produced from cfndsl needs to be altered.

Syntax proposal

CfhighlanderTemplate do
   
     Name 'myapp'

     Component template:'vpc', cfinline: true
     Component template: 'asg', cfinline: true
     Component template: 'alb', cfinline: true
  
end

Feature request: get diff output with new component versions

When changing a component's version, we have no mechanism to verify that the compiled output will also be changed. For small changes to components, it's usually easy for a developer to verify that the new code is safe to deploy, simply by visiting GitHub. But for large changes, verifying can involve more lengthy processes, such as using diff, updates to CloudFormation, etc, which can be inefficient and error prone.

It would be nice to have an option to cfcompile that would allow us to see the diff if we were to use a new component's version in the project. For example, a command like
cfhighlander cfcompile --diff vpc1.0.0=vpc1.0.1.

Note that this is separate from showing the actual diff of the components themselves -- this covers only the generated output.

This could also serve as a testing tool, as we can verify that a config can work across multiple versions of a particular component, and vice versa.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.