linki / cloudformation-operator

A Kubernetes operator for managing CloudFormation stacks via a CustomResource

cloudformation-operator

A Kubernetes operator for managing CloudFormation stacks via kubectl and a custom resource definition.

Warning: this project is in alpha state. It should only be used to try out the demo and get the general idea.

This version uses the new operator-sdk. It's untested and may not work correctly

Deploy to a cluster

You need API access to a cluster running at least Kubernetes v1.17.0+.

Start the CloudFormation operator in your cluster by using the provided manifests:

$ make deploy IMG=quay.io/linki/cloudformation-operator:latest

Modify the region flag to match your cluster's.

Additionally you need to make sure that the operator Pod has enough AWS IAM permissions to create, update and delete CloudFormation stacks as well as permission to modify any resources that are part of the CloudFormation stacks you intend to deploy. In order to follow the example below it needs access to CloudFormation as well as S3.

The operator will require an IAM role or user credentials. Use the following Policy document as a guideline in order to follow the tutorial:

MyIAMRole:
  Properties:
    ...
    Policies:
    - PolicyDocument:
        Statement:
        - {Action: 'cloudformation:*', Effect: Allow, Resource: '*'}
        - {Action: 's3:*', Effect: Allow, Resource: '*'}
        Version: '2012-10-17'
    ...

The operator will usually use the IAM role of the EC2 instance it's running on, so you have to add those permissions to that role. If you're using Kube2IAM or similar and give your Pod a dedicated IAM role then you have to add the permissions to that role.

Once running the operator should print some output but shouldn't actually do anything at this point. Leave it running, keep watching its logs and continue with the steps below.

Demo

Create stack

Currently you don't have any stacks.

$ kubectl get stacks
No resources found.

Let's create a simple one that manages an S3 bucket:

apiVersion: cloudformation.linki.space/v1alpha1
kind: Stack
metadata:
  name: my-bucket
spec:
  template: |
    ---
    AWSTemplateFormatVersion: '2010-09-09'

    Resources:
      S3Bucket:
        Type: AWS::S3::Bucket
        Properties:
          VersioningConfiguration:
            Status: Suspended

The Stack resource's definition looks a lot like any other Kubernetes resource manifest. The spec section describes an attribute called template which contains a regular CloudFormation template.

Go ahead and submit the stack definition to your cluster:

$ kubectl apply -f config/samples/cfs-my-bucket-v1.yaml
stack "my-bucket" created
$ kubectl get stacks
NAME        AGE
my-bucket   21s

Open your AWS CloudFormation console and find your new stack.

Once the CloudFormation stack is created check that your S3 bucket was created as well.

The operator will write back additional information about the CloudFormation Stack to your Kubernetes resource's status section, e.g. the stackID:

$ kubectl get stacks my-bucket -o yaml
spec:
  template:
  ...
status:
  stackID: arn:aws:cloudformation:eu-central-1:123456789012:stack/my-bucket/327b7d3c-f27b-4b94-8d17-92a1d9da85ab

Voilà, you just created a CloudFormation stack by only talking to Kubernetes.

Update stack

You can also update your stack: Let's change the VersioningConfiguration from Suspended to Enabled:

apiVersion: cloudformation.linki.space/v1alpha1
kind: Stack
metadata:
  name: my-bucket
spec:
  template: |
    ---
    AWSTemplateFormatVersion: '2010-09-09'

    Resources:
      S3Bucket:
        Type: AWS::S3::Bucket
        Properties:
          VersioningConfiguration:
            Status: Enabled

As with most Kubernetes resources you can update your Stack resource by applying a changed manifest to your Kubernetes cluster or by using kubectl edit stack my-stack.

$ kubectl apply -f config/samples/cfs-my-bucket-v2.yaml
stack "my-bucket" configured

Wait until the operator discovered and executed the change, then look at your AWS CloudFormation console again and find your stack being updated, yay.

Tags

You may want to assign tags to your CloudFormation stacks. The tags added to a CloudFormation stack will be propagated to the managed resources. This feature may be useful in multiple cases, for example, to distinguish resources at billing report. Current operator provides two ways to assign tags:

• --tag command line argument or AWS_TAGS environment variable which allows setting default tags for all resources managed by the operator. The format is --tag=foo=bar --tag=wambo=baz on the command line or with a line break when specifying as an env var. (e.g. in zsh: AWS_TAGS="foo=bar"$'\n'"wambo=baz")
• tags parameter at kubernetes resource spec:

apiVersion: cloudformation.linki.space/v1alpha1
kind: Stack
metadata:
  name: my-bucket
spec:
  tags:
    foo: dataFromStack
  template: |
    ---
    AWSTemplateFormatVersion: '2010-09-09'

    Resources:
      S3Bucket:
        Type: AWS::S3::Bucket
        Properties:
          VersioningConfiguration:
            Status: Enabled

Resource-specific tags have precedence over the default tags. Thus if a tag is defined at command-line arguments and for a Stack resource, the value from the Stack resource will be used.

If we run the operation and a Stack resource with the described above examples, we'll see such picture:

Parameters

However, often you'll want to extract dynamic values out of your CloudFormation stack template into so called Parameters so that your template itself doesn't change that often and, well, is really a template.

Let's extract the VersioningConfiguration into a parameter:

apiVersion: cloudformation.linki.space/v1alpha1
kind: Stack
metadata:
  name: my-bucket
spec:
  parameters:
    VersioningConfiguration: Enabled
  template: |
    ---
    AWSTemplateFormatVersion: '2010-09-09'

    Parameters:
      VersioningConfiguration:
        Type: String
        Default: none
        AllowedValues:
        - "Enabled"
        - "Suspended"

    Resources:
      S3Bucket:
        Type: AWS::S3::Bucket
        Properties:
          VersioningConfiguration:
            Status:
              Ref: VersioningConfiguration

and apply it to your cluster:

$ kubectl apply -f config/samples/cfs-my-bucket-v3.yaml
stack "my-bucket" configured

Since we changed the template a little this will update your CloudFormation stack. However, since we didn't actually change anything because we injected the same VersioningConfiguration value as before, your S3 bucket shouldn't change.

Any CloudFormation parameters defined in the CloudFormation template can be specified in the Stack resource's spec.parameters section. It's a simple key/value map.

Outputs

Furthermore, CloudFormation supports so called Outputs. These can be used for dynamic values that are only known after a stack has been created. In our example, we don't define a particular S3 bucket name but instead let AWS generate one for us.

Let's change our CloudFormation template to expose the generated bucket name via an Output:

apiVersion: cloudformation.linki.space/v1alpha1
kind: Stack
metadata:
  name: my-bucket
spec:
  parameters:
    VersioningConfiguration: Enabled
  template: |
    ---
    AWSTemplateFormatVersion: '2010-09-09'

    Parameters:
      VersioningConfiguration:
        Type: String
        Default: none
        AllowedValues:
        - "Enabled"
        - "Suspended"

    Resources:
      S3Bucket:
        Type: AWS::S3::Bucket
        Properties:
          VersioningConfiguration:
            Status:
              Ref: VersioningConfiguration

    Outputs:
      BucketName:
        Value: !Ref 'S3Bucket'
        Description: Name of the sample Amazon S3 bucket.

Apply the change to our cluster and wait until the operator has successfully updated the CloudFormation stack.

$ kubectl apply -f config/samples/cfs-my-bucket-v4.yaml
stack "my-bucket" configured

Every Output you define will be available in your Kubernetes resource's status section under the outputs field as a key/value map.

Let's check the name of our S3 bucket:

$ kubectl get stacks my-bucket -o yaml
spec:
  template:
  ...
status:
  stackID: ...
  outputs:
    BucketName: my-bucket-s3bucket-tarusnslfnsj

In the template we defined an Output called BucketName that should contain the name of our bucket after stack creation. Looking up the corresponding value under .status.outputs[BucketName] reveals that our bucket was named my-bucket-s3bucket-tarusnslfnsj.

Delete stack

The operator captures the whole lifecycle of a CloudFormation stack. So if you delete the resource from Kubernetes, the operator will teardown the CloudFormation stack as well. Let's do that now:

$ kubectl delete stack my-bucket
stack "my-bucket" deleted

Check your CloudFormation console once more and validate that your stack as well as your S3 bucket were deleted.

Command-line arguments

Argument	Environment variable	Default value	Description
assume-role			Assume AWS role when defined. Useful for stacks in another AWS account. Specify the full ARN, e.g. arn:aws:iam::123456789:role/cloudformation-operator
capability			Enable specified capabilities for all stacks managed by the operator instance. Current parameter can be used multiple times. For example: --capability CAPABILITY_NAMED_IAM --capability CAPABILITY_IAM. Or with a line break when specifying as an environment variable: AWS_CAPABILITIES=CAPABILITY_IAM$'\n'CAPABILITY_NAMED_IAM
dry-run			If true, don't actually do anything.
tag ...			Default tags which should be applied for all stacks. The format is --tag=foo=bar --tag=wambo=baz on the command line or with a line break when specifying as an env var. (e.g. in zsh: AWS_TAGS="foo=bar"$'\n'"wambo=baz")
namespace	WATCH_NAMESPACE	default	The Kubernetes namespace to watch
region			The AWS region to use

Cleanup

Clean up the resources:

$ make undeploy

Build and run locally

This project uses the operator sdk.

$ make
$ WATCH_NAMESPACE=default KUBERNETES_CONFIG=~/.kube/config make run OPERATOR_FLAGS="--region eu-central-1"

Build the docker image

$ make docker-build quay.io/linki/cloudformation-operator:latest
$ make docker-push quay.io/linki/cloudformation-operator:latest

Test it locally

You can use OPERATOR_FLAGS to pass in flags using the operator-sdk.

Assuming you are using minikube:

$ minikube start # you will be have a kubeconfig read to use by cloudformation operator
$ export AWS_PROFILE=my_profile # setup your aws config
$ cd $GOPATH/src/github.com/linki/cloudformation-operator
$ # run cloudformation operator based on previous settings and env vars
$ WATCH_NAMESPACE=staging make run OPERATOR_FLAGS="--dry-run=true --region=eu-central-1"
I0122 16:31:14.509064  195514 request.go:645] Throttling request took 1.027790903s, request: GET:https://api.crc.testing:6443/apis/template.openshift.io/v1?timeout=32s
2021-01-22T16:31:15.863-0500    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2021-01-22T16:31:15.864-0500    INFO    setup   
2021-01-22T16:31:15.864-0500    INFO    setup   starting manager
2021-01-22T16:31:15.864-0500    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2021-01-22T16:31:15.864-0500    INFO    controller-runtime.manager.controller.stack     Starting EventSource    {"reconciler group": "cloudformation.linki.space", "reconciler kind": "Stack", "source": "kind source: /, Kind="}
2021-01-22T16:31:15.965-0500    INFO    controller-runtime.manager.controller.stack     Starting Controller     {"reconciler group": "cloudformation.linki.space", "reconciler kind": "Stack"}
2021-01-22T16:31:15.965-0500    INFO    controller-runtime.manager.controller.stack     Starting workers        {"reconciler group": "cloudformation.linki.space", "reconciler kind": "Stack", "worker count": 1}

cloudformation-operator's Issues

Report CloudFormation events back to Stack resource

Events from AWS especially when CF stack creation failed should be surfaced to the Stack CRD via events or something similar.

Is it maintained?

And is it a personal project, or used in a professional context?

It looks nice, and trying to understand the level of effort that will be put there later on.
If I use it, I'll obviously help you then. (But it is not clear yet the direction we are taking in our company).

Basically, we need to provide RDS to our multitenant openshift cluster, and provide day 2 operation as well.

Thanks for your insight!

related to sorenmat/k8s-rds#44

tolerations issue in Helm

Hi,
Under helm/c-o/templates/deployment.yaml, the tolerations section should be under spec (currently it's under containers), pls kindly help fix.

Also, is it possible to open the permission so we could create branches and raise PR?

Cheers,

Add additonal owner Tag denoting the owning cluster / operator instance

Unable to read the CFT Stack

I can see the stack getting created,

But when i'm trying to access it in the AWS Console i don't see any available stacks.

Export Stack resource labels to Cloudformation Stacks

Add documentation regarding related projects

Add some notes regarding similar projects:

• AWS's Service Operator
• Google's Config Connector

allow duplicate stack names across Kubernetes namespaces

RBAC

When deploying to a Kubernetes cluster with RBAC enabled, the follow error occurs in the cloudformation-operator logs:

time="2018-03-27T06:36:28Z" level=fatal msg="stacks.cloudformation.linki.space is forbidden: User \"system:serviceaccount:default:default\" cannot list stacks .cloudformation.linki.space at the cluster scope"

Will send a pull request shortly to add an RBAC policy to the repo/readme.

Generate a config map with the CloudFormation outputs

It would be nice if this could generate/Manage a K8s configMap (or a k8s secret) with all the outputs
of the cloudFormation (instead of only keeping those in the "status" section)

This would permit to mount directly the config map to a deployment for example and help automatically "glue" things.

---

I know k8s doesn't support auto-"reloading" services that consumes a configMaps that has changed, but even knowing that, it's would still be a real nice feature...

Add Readme

Allow template content to be fetched from configmap or env

The template for the Cloudformation yaml is currently inline string and difficult to maintain. Can we allow template to be fetched from config map or an env variable please. This will allow separating the stack yaml with cloud formation. Thanks

operator-sdk supports allow pass in flags

The operator-sdk has recently added the pass in flags feature via operator-framework/operator-sdk#264 on the master branch. Feel free to check that out.

Use paginated API calls

Add support for Cloudformation "detect stack drifts"

There is a nice Cloudformation feature called "detect stack drifts" which allow us to see if the resources deployed by CF drifted from the CF definition's state (like if someone changed an s3 option by hand/cli).

How i see it working for this operator?

• We could have an option in the k8s stack resource that looks like:

  ensureNoDrift: true|false
  driftCheckFrequency: "24h"

A parallel "worker" would trigger CF drift checks on time.

• it would then queue in CF a "detect drift request" and poll for it's results.
• If a drift is detected, it would trigger an update of the CF using the previous definition (only if no other operation is already in progress updates or something else)

How to remove a stack created with cloudformation-operator ?

Hi, can this be used to delete the stacks besides creating them? If so, can you please guide me? I couldn't find anything online nor in this repo.

That's how we create a stack and I would like to remove it when a speific condition is met.

&cloudformation.Stack{
		TypeMeta:   TypeMeta("cloudformation.linki.space/v1alpha1", "Stack"),
		ObjectMeta: i.objectMeta,
		Spec: cloudformation.StackSpec{
			Parameters: i.Parameters,
			Tags:       i.Tags,
			Template:   i.Template,
		},
	}

adopt the operator framework

There's a work in progress in this fork 😄

Add support for Google Cloud Deployment Manager

Dependabot can't parse your go.mod

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: github.com/operator-framework/[email protected] requires
	vbom.ml/[email protected]: unrecognized import path "vbom.ml/util" (https fetch: Get https://vbom.ml/util?go-get=1: EOF)

View the update logs.