aws-controllers-k8s / community

AWS Controllers for Kubernetes (ACK) is a project enabling you to manage AWS services from Kubernetes

AWS Controllers for Kubernetes (ACK)

AWS Controllers for Kubernetes (ACK) lets you define and use AWS service resources directly from Kubernetes. With ACK, you can take advantage of AWS managed services for your Kubernetes applications without needing to define resources outside of the cluster or run services that provide supporting capabilities like databases or message queues within the cluster.

ACK is an open source project built with ❤️ by AWS. The project is composed of many source code repositories containing a common runtime, a code generator, common testing tools and Kubernetes custom controllers for individual AWS service APIs.

IMPORTANT Please be sure to read our documentation about release versioning and maintenance phases and note that ACK service controllers in the Preview maintenance phase are not recommended for production use. Use of ACK controllers in Preview maintenance phase is subject to the terms and conditions contained in the AWS Service Terms, particularly the Beta Service Participation Service Terms, and apply to any service controllers in a Preview maintenance phase.

• Overview
• Getting Started
• Help & Feedback
• Contributing
• License

Overview

Kubernetes applications often require a number of supporting resources like databases, message queues, and object stores. AWS provides a set of managed services that you can use to provide these resources for your apps, but provisioning and integrating them with Kubernetes was complex and time consuming. ACK lets you define and consume AWS services and resources directly from a Kubernetes cluster. It gives you a unified way to manage your application and its dependencies.

ACK is a collection of Kubernetes custom resource definitions (CRDs) and custom controllers working together to extend the Kubernetes API and manage AWS resources on your behalf.

Getting Started

Please see the list of ACK service controllers currently in one of our project stages.

You can install any of the controllers in the RELEASED project stage using Helm (recommended) or manually using the raw Kubernetes manifests contained in the individual ACK service controller's source repository.

Once installed, Kubernetes users may apply a custom resource (CR) corresponding to one of the resources exposed by the ACK service controller for the service.

To view the list of custom resources and each CR's schema, visit our reference documentation.

Help & Feedback

For help, please consider the following venues (in order):

• ACK project documentation
• Search open issues
• File an issue
• Chat with us on the #aws-controllers-k8s channel in the Kubernetes Slack community.

Contributing

We welcome community contributions and pull requests.

See our contribution guide for more information on how to report issues, set up a development environment, and submit code.

We adhere to the Amazon Open Source Code of Conduct.

You can also learn more about our Governance structure.

Community Meeting

ACK Community meeting is held every week. Everyone is welcome to participate.

Details

• Agenda/Notes: link
  • Notes from each meeting are captured here.
• When: every Thursday at 9:00 AM [PST][pst-timezone]
• Where: Zoom meeting

License

This project is licensed under the Apache-2.0 License.

community's Issues

Aurora Serverless Service Controller

ACK Service Controller

Support for Aurora Serverless

Checklist for Dev Preview

• Code generation (make build-controller)
• End-to-end test (make kind-test)
• Docs are updated, at least:
  • testing
  • svc controller listing
• Container image in public registry

MVP

In order to validate the design and gain understanding of the shortcomings, we will put together an MVP. This SHOULD cover an end-to-end implementation of the business logic based on #15 for three services (S3, IAM, DynamoDB) incl. install procedure and testing.

The MVP will depend on #3 #4 #5 #6 and #14.

Allow for many parallel reconciliations of similar resources without frequently hitting API limits

Describe the solution you'd like
To avoid issues at scale, it would be great if aws-service-operator-k8s could attempt to batch similar requests that are occurring in parallel over a short period of time and batch those requests to avoid, for example, parallel reconciliation of 50 ec2 Instances from making 50 separate describe calls to the API.

It should be possible through the use of an out-of-band process or thread to queue similar requests for a brief period of time such that they can be batched into fewer AWS API calls, allowing for less of a chance of bumping up against API limits.

Depending on the purpose of the call, it may also be beneficial to allow for results to also be cached for a certain period of time. This might not be desirable for the purpose of periodic reconciliation of a resource, but it would likely help avoid making calls to the API for the purposes of dependency management for example, especially in the case of trying to reconcile a dependent object shortly after a resource is created and within the eventual consistency period for the resource.

Describe alternatives you've considered
A description of any alternative solutions or features you've considered.

Determine single or multiple binaries

We have a couple options when it comes to building the application binaries:

• Single application binary with all service controllers
• Multiple application binaries with individual service controllers

Single application binary

The approach here is to compile all service controllers into a single application binary. Each service controller will run as a separate Go routine that watches for CRs of types associated with its AWS service API.

Pros:

• Simple installation. One Helm chart that installs a single image as a Deployment.
• Simple versioning and upgrade. As simple as one call to helm upgrade
• Simple operation. Since it's a single Pod definition, logging and monitoring is inherently easier.

Cons:

• No fine-grained upgrade story. If you want to only upgrade the S3 service controller, you need to update the one image and redeploy the Pod(s)
• If we use IRSA/kIAM/kube2iam for managing a single IAM role for the AWS service permissions, this single IAM role would essentially need superuser privileges on all services. If that is viewed as unacceptable, we'll need to use a technique of using STS::AssumeRole along with a ConfigMap+Secret to map individual AWS service controllers to service-scoped IAM roles.

Multiple application binaries

Here, the solution is to compile service controllers into individual application binaries, one for each AWS service API.

Pros:

• Fine-grained installation. Users who want to create only S3 buckets would only install the S3 controller binary
• Fine-grained versioning and upgrade. Users who only want to upgrade to the latest S3 service controller would be able to do that and not impact other service controllers running in the cluster.
• Ability to use IRSA/kIAM/kube2iam with service-scoped IAM role, removing a need for a complex ConfigMap+Secret solution to IAM.

Cons:

• More operational burden. Potentially could have hundreds of these individual service controller binaries in the cluster, each requiring their own logging config, monitoring, etc.
• Much greater versioning complexity. There would need to be a Helm chart per service controller in order to separately version each individual service controller image. Keeping track of these individual service controller versions could prove tricky.
• Many more IAM roles to manage and audit.

Name cleaning produces "MaxIDleConnectionsPercent"

Found while generating the RDS API types:

type ConnectionPoolConfigurationInfo struct {
	ConnectionBorrowTimeout *int64 `json:"connectionBorrowTimeout,omitempty"`
	InitQuery *string `json:"initQuery,omitempty"`
	MaxConnectionsPercent *int64 `json:"maxConnectionsPercent,omitempty"`
	MaxIDleConnectionsPercent *int64 `json:"maxIDleConnectionsPercent,omitempty"`
	SessionPinningFilters []*string `json:"sessionPinningFilters,omitempty"`
}

Note the misspelled MaxIDleConnectionsPercent field...

Kubernetes native way to consume CustomResources

Default functionality in K8s is to contact an Application by its Service DNS. To Contact AWS resources created in K8s it would be also great to have this native like functionality, If I own service A and it requires S3 bucket B which is created together in same namespace it would be great to be able to just point to bucket B and the DNS takes care of translating it to aws url. Have you thought about such feature?

The other alternative which is easy to implement and I assume that would be first iteration is that parameters of the AWS object are just added to the Custom Resource but to use such object then in an application the application must be K8s aware.

Support IRSA for the service operator role assumption

Is your feature request related to a problem?
The old version of this operator did not natively support using IAM Roles for Service Accounts to assume the role necessary to provision resources.

Describe the solution you'd like
I think all that needs to happen is that the service operator server gets built with a compatible version of the aws-sdk-go, i.e. version 1.23.13 or later. Then, a method for adding the proper annotation to the ServiceAccount resource in the manifests needs to be provided.

Possibility of "adopting" existing AWS resources

Cloudformation supports importing existing resources https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html

A parallel issue is opened in aws/aws-cdk-rfcs#52

Incorrect Snake cases for normalized names

For some initialisms and acronyms, snake casing is producing incorrect output. Snake case is used primarily to generate clean package names for resources.

An example problematic initialism is "Uri", which is part of the word "Security". The normalization process in pkg/names/names.go for snake-casing the name "CacheSecurityGroup" is outputting "cache_sec_uri_ty_group" instead of "cache_security_group". Fun!

Consistent Tags resource handling

Different AWS service API use different data types to represent pretty much the exact same thing: a set of Key:Value pairs ("tags") for the resource. For instance, some APIs use a map[string]string as the underlying data type in API payloads. Some use a []struct{string, string}. Some use map[string]struct{}`, etc etc.

In addition to inconsistent data type representation of tags, the APIs also have inconsistent methods of adding, removing, replacing, and querying for tags on a resource. For example, some APIs have a TagResource and UntagResource API call. Others have an AddTag or RemoveTag call. Some APIs allow tagging a resource at create time. Others only allow updating tags after creation.

We want to make the experience of tagging and untagging resources consistent across all AWS APIs -- both from a data type representation as well as the behaviour of set/unset methods.

For all CRDs exposed by an ACK service controller, we want to have a consistent Spec.Tags field:

type {Resource}Spec struct {
    // Tags are a collection of string key/value pairs indicating the AWS
    // Tags that should be associated with the resource
    Tags map[string]string
    // Rest of Spec fields...
}

There will, of course, be no TagResource or UntagResource Kubernetes API call, since Kubernetes API is declarative and any changes to the desired state of a resource are simply handled by kubectl apply'ing the new desired resource state (as the CR's Spec struct).

So, the ACK service controller will need to have logic embedded in it that essentially instructs the service controller to handle tag information set/unset/add/remove logic in the way that the AWS service API for that controller expects. So, if the AWS service API has a TagResources API call and the only thing about a resource's desired state that has changed is the tags collection, then the service controller would call the TagResources API call, etc.

Ambiguity in generated sdk.go file for controllers.

Describe the bug
Some of the autogenerated code throws error when building controller's main.go

Steps to reproduce

• Add '_ "github.com/aws/aws-controllers-k8s/services/apigatewayv2/pkg/resource/api"' to main.go file so that resourceManagerFactory is initialized.
• run go build -o ../../bin/controller main.go
• Following errors show up apart from #98

# github.com/aws/aws-controllers-k8s/services/apigatewayv2/pkg/resource/api
../../pkg/resource/api/sdk.go:61:3: undefined: tmpElem0
../../pkg/resource/api/sdk.go:61:36: elem0.ImportInfo undefined (type *string has no field or method ImportInfo)
../../pkg/resource/api/sdk.go:62:23: undefined: tmpElem0
../../pkg/resource/api/sdk.go:67:3: undefined: tmpElem1
../../pkg/resource/api/sdk.go:67:36: elem1.Warnings undefined (type *string has no field or method Warnings)
../../pkg/resource/api/sdk.go:68:23: undefined: tmpElem1
../../pkg/resource/api/sdk.go:108:3: undefined: tmpElem0
../../pkg/resource/api/sdk.go:108:36: elem0.ImportInfo undefined (type *string has no field or method ImportInfo)
../../pkg/resource/api/sdk.go:108:36: too many errors

• iteration should be performed on responseObject and not Ko.Status.ImportInfo

tmp0 := []*string{}
	for _, elem0 := range ko.Status.ImportInfo {
		tmpElem0.__listOf__string = elem0.ImportInfo
		tmp0 = append(tmp0, tmpElem0)
	}
	ko.Status.ImportInfo = tmp0

Expected outcome
Controller's main.go to build successfully

Environment

• Kubernetes version 1.14
• Using EKS (yes/no), if so version? yes
• AWS service targeted (S3, RDS, etc.) apigatewayv2

Pull in AWS API call throttling into common runtime

We're gonna need a common approach for throttling AWS API calls.

The AppMesh Controller for Kubernetes has an AppMesh API-specific throttler library:

https://github.com/aws/aws-app-mesh-controller-for-k8s/tree/master/pkg/aws/throttle

Let's pull that into pkg/throttle in ACK and make it generically-configurable for any AWS service API

Allow overriding target AWS Region for CRs

AWS resources are created in a specific region. When calling an AWS service API, a regional API endpoint/URI is used. Similarly, when instantiating an aws-sdk-go Session, a regional endpoint is determined by looking at the AWS_REGION environs variable.

Currently, ACK CRs are created in the AWS Region associated with the AWS_REGION environs variable for the Pod running the ACK service controller.

We want to support a Kubernetes user being able to specify an AWS Region in which to create a particular resource. I'm thinking about support for a standardized services.k8s.aws/region annotation that can be associated with a Namespace (providing a default AWS region for CRs within that Namespace to use to override the default one used by the service controller) as well as the ability for a Kubernetes user to specify a service.k8s.aws/region annotation directly on the CR itself to override the Namespace default.

Create IAM role for a pod and associating it with the pod service account via IAM OIDC

Is your feature request related to a problem?
I have a pipeline that deploys a Kubernetes pod to a cluster. I want that pod to have an IAM role, to interact with AWS resources. Currently theres no way (that I know of, I'd love to be wrong) to both create an IAM role and associate with the pod in the same deployment manifest file

Describe the solution you'd like
An IAM operator that exports the IAM role, to be used in the other part of the manifest for associating with the pod

If anyone can think of another way to achieve this, please let me know

Operator Hub update

Update https://operatorhub.io/operator/aws-service with the new info.

Operator is unable to find the S3 bucket

Describe the bug
After creating the aws operator using the aws-service-operator.yaml and isnerting the correct flags,
the logs show that there is no such bucket.

Steps to reproduce

1. Download aws-service-operator.yaml from the AWS page
2. Insert args:
   • args:
     • server
     • --cluster-name=
     • --region=
     • --account-id=
     • --bucket=
     • --loglevel=Debug
3. kubectl apply -f aws-service-operator -n
4. kubectl -f apply <elasticache.yaml>

Expected outcome
The above will create the cdfn's and the namespace
Afterwords it will create an elasticache service that will send an API request to Cloudformation
to create a stack that will create a Redis cluster

Environment

• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-18T14:56:51Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.12-eks-c500e1", GitCommit:"c500e11584c323151d6ab17526d1ed7461e45b0c", GitTreeState:"clean", BuildDate:"2019-10-22T03:11:52Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

• Using EKS (yes), if so version? :

eks.6

• AWS service targeted (S3, RDS, etc.):

Cloudformation/S3/Redis

*Logs from the aws-service-operator deploy:

time="2019-12-02T13:50:02Z" level=error msg="error creating elasticache 'jenkins-integ-test-20191202014223'" error="ValidationError: S3 error: The specified bucket does not exist\nFor more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html\n\tstatus code: 400, request id: c5406494-b8a3-4043-ab7f-eed653e99a20" hostname=aws-service-operator-5459ffd49b-pw9m5

Handling of AWS resource dependencies

The ideal resources that fit the k8s model do not depend on the order in which they are created (e.g. a backend component should wait until database is ready etc.), however some AWS resources depend on the creation order. Cloudformation has various mechanisms to handle dependencies like Outputs, Fn::ImportValue https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-crossstackref.html or DependsOn https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-dependson.html

How dependencies between AWS resources will be handled by ASO?

Cross account resource management (CARM)

This issue is about AWS accounts management within a single Kubernetes cluster.

We want to:

• Allow users to securely create resources across multiple AWS accounts without having to own/manage multiple Kubernetes clusters.
• Help users maintain/administer controllers access to stored AWS accounts.
• Limit controllers permissions when calling AWS API.
• Prevent unauthorized users from making CRUD operations on resources on a different AWS account.

Link to CARM design proposal

Related to: #3 and #25

Determine service support for tag-on-create

Determine which services support tag-on-create semantics and for which resources in those services. If we are able to use tag-on-create, we might be able to use tag-based authz to limit the blast radius of the IAM role under which the controller's pod service account executes.

Business logic

This design issue is about how the business logic of ASO custom controllers is implemented. The goal is to find a scalable and flexible way to manage AWS services. Usage of requirements key words as per RFC 2119.

Scope

In scope for this issue is the question how the business logic of ASO custom controllers is implemented, that is, how the management of AWS services is achieved. Out of scope for this issue is the selection of the custom controller framework, see #4 for this.

Input

Relevant technologies that could be used, either on their own or in combination:

• The AWS SDK for Go
• The AWS Cloud Development Kit
• AWS CloudFormation
• Terraform
• Validating Webhooks

Input from roadmap issue aws/containers-roadmap#456:

• @whereisaaron in:
  • aws/containers-roadmap#456 (comment)
  • aws/containers-roadmap#456 (comment)
• @daviddyball in:
  • aws/containers-roadmap#456 (comment)
  • aws/containers-roadmap#456 (comment)
  • aws/containers-roadmap#456 (comment)
• @cpoole in aws/containers-roadmap#456 (comment)
• @iAnomaly in aws/containers-roadmap#456 (comment)
• @sftim in aws/containers-roadmap#456 (comment)

Requirements

1. It MUST be possible to write business logic in Go.
2. The implementation details of the business logic implementation SHOULD NOT leak to the ASO client.
3. All AWS services MUST be supported; where not possible, a mechanism MUST exist to provide a custom support.

Document deployment and upgrade approach for ACK itself

Currently, we think that the ASO server implementation will be entirely stateless. There may be caches, of course, but the state that ASO considers will be stored in the Kubernetes etcd backend and the AWS services themselves.

So, the upgrade process for ASO itself should be relatively simple: just increment the image version/tag for the Deployment YAML that describes the ASO application in the user's Kubernetes cluster.

However, over time, as AWS services evolve, so will the CRD(s) that describe(s) the object(s) the service exposes. We should document how we will handle evolving the CRD schema over time, how we will approach rolling out new CRD versions and how we will ensure that an ASO controller can read and reconcile CRs with different CRD schema versions.

A variety of options:

• Standardize on Helm and keep deployment artifacts only in the eks Helm charts repository
• Use a Kustomize config directory in the aws/aws-service-operator-k8s source repository and have a script that automatically creates/updates a Helm chart for ASO in the eks-charts Git repo

Semantics of destructive operations

If an ACK user deletes a Kubernetes namespace that has, say, an S3 bucket custom resource in it, is the S3 bucket also deleted or not?

The answer to this question (and the respective UX) should follow the "no surprises" principle. IOW: opting into (forcing) destructive operations rather than silently cascading the delete from the Kubernetes cluster perimeter to AWS services.

failure generating RDS API

Describe the bug

When trying to ack-generate apis for RDS, running into the following:

jaypipes@thelio:~/go/src/github.com/aws/aws-service-operator-k8s$ go run cmd/ack-generate/main.go apis rds < /tmp/rds.yaml
Error: failed to find DBClusterOptionGroupMembership when looking up arrayTypeName DBClusterOptionGroupMemberships

A recent commit changed the Go type introspection to return an error instead of returning "!! UNKNOWN ARRAY MEMBER TYPE !!". Apparently there's something about RDS' DBClusterOptionGroupMembership that the introspection code doesn't like.

Test apigatewayv2 controller locally

• Run the apigateway controller locally and do manual testing.
• Create followup issues for the bugs found in the code.

Custom controller framework

This design issue is about how custom controllers are written in ASO. The goal is to select a widely used framework that has a low barrier to contributing to ASO. Usage of requirements key words as per RFC 2119.

Scope

In scope for this issue is the selection of the custom controller framework, that is, a Go-based framework supporting the creation of custom controllers used in ASO. Out of scope for this issue is the question how the business logic (managing AWS services) in the custom controllers is implemented, see #5 for this.

Input

Relevant candidates:

• Kubebuilder
• Operator SDK

Input from roadmap issue aws/containers-roadmap#456:

• @daviddyball in aws/containers-roadmap#456 (comment)

Requirements

1. The custom controller framework MUST support Go.
2. The custom controller framework SHOULD be well-documented, well-maintained, and allow for easy on-boarding.

Rename repository to aws-controllers-k8s

We are changing the repository name to match the finalized name for the project, which is AWS Controllers for Kubernetes, or ACK.

This will require quite some coordination, so I'm creating an issue to track this. Because of the Golang imports pointing to aws/aws-service-operator-k8s, we will need to do a single PR that changes all the import paths and references to the old aws-service-operator name.

Improper import path generated for SQS controller's `types.go`

When attempting to generate the SQS controller, running into the following:

jaypipes@thelio:~/go/src/github.com/aws/aws-controllers-k8s$ ./scripts/build-controller.sh sqs
Building Kubernetes API objects for sqs
Generating deepcopy code for sqs
-: types.go:18:5: invalid import path: "map[string]*sqs"

looking at the generated code:

jaypipes@thelio:~/go/src/github.com/aws/aws-controllers-k8s$ cat services/sqs/apis/v1alpha1/types.go 
// Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"). You may
// not use this file except in compliance with the License. A copy of the
// License is located at
//
//     http://aws.amazon.com/apache2.0/
//
// or in the "license" file accompanying this file. This file is distributed
// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
// express or implied. See the License for the specific language governing
// permissions and limitations under the License.

// Code generated by ack-generate. DO NOT EDIT.

package v1alpha1
import (
    "map[string]*sqs"

)
...

pretty obviously I'm not generating the types.go file imports properly...

Is this project still active ?

Hi,

Is there an ETA for this operator ? It would be great if we can have an officially supported operator for AWS services. Looking forward for it !

Project logo

Support for creating Network Services

Is your feature request related to a problem?
We plan to use AWS Operator for provisioning Applications that need network resources, things like
VPCs, Internet Gateway, Subnets

Describe the solution you'd like
The current CRD/Cloud Formation pattern is fine for creating these and two patterns for consuming these would be the ConfigMap or API query of resource Status, the later preferred since a higher level Operator will drive the overall flow and resolve state.

Describe alternatives you've considered
Fork current AWS Operator project or write a new operator for this feature.

Array types not handled properly in `ack-generate apis`

While generating CRD type definitions for some APIs, I've noticed the following in some of the generated structs:

type ImageScanFindings struct `aws:"ImageScanFindings"` {
	FindingSeverityCounts map[string]string `json:"findingSeverityCounts,omitempty" aws:"findingSeverityCounts"`
	Findings []!!! UNKNOWN ARRAY OBJECT TYPE !!! `json:"findings,omitempty" aws:"findings"`
	ImageScanCompletedAt string `json:"imageScanCompletedAt,omitempty" aws:"imageScanCompletedAt"`
	VulnerabilitySourceUpdatedAt string `json:"vulnerabilitySourceUpdatedAt,omitempty" aws:"vulnerabilitySourceUpdatedAt"`
}

when I take a look at the OpenAPI3 Schema for the API (in the above case, the ECR API), I find this:

    ImageScanFindings:
      properties:
        findingSeverityCounts:
          $ref: '#/components/schemas/FindingSeverityCounts'
        findings:
          $ref: '#/components/schemas/ImageScanFindingList'
        imageScanCompletedAt:
          $ref: '#/components/schemas/ScanTimestamp'
        vulnerabilitySourceUpdatedAt:
          $ref: '#/components/schemas/VulnerabilitySourceUpdateTimestamp'
      type: object

You will note that the findings property points to an ImageScanFindlingList schema, which looks like this:

    ImageScanFindingList:
      items:
        properties:
          attributes:
            $ref: '#/components/schemas/AttributeList'
          description:
            $ref: '#/components/schemas/FindingDescription'
          name:
            $ref: '#/components/schemas/FindingName'
          severity:
            $ref: '#/components/schemas/FindingSeverity'
          uri:
            $ref: '#/components/schemas/Url'
        type: object
      type: array

the above is an array of object with a set of properties (attributes, description, name, severity, and uri). Not coincidentally, that object schema is the same as ImageScanFinding's schema:

    ImageScanFinding:
      properties:
        attributes:
          $ref: '#/components/schemas/AttributeList'
        description:
          $ref: '#/components/schemas/FindingDescription'
        name:
          $ref: '#/components/schemas/FindingName'
        severity:
          $ref: '#/components/schemas/FindingSeverity'
        uri:
          $ref: '#/components/schemas/Url'
      type: object

For some reason, the code that handles array type processing in the type definition generators isn't properly understanding that the ImageScanFindings.findings field should be an array of type pointer to ImageScanFinding.

Need to convert metav1.Time to time.Time when setting SDK struct fields

Generated the ElastiCache service controller and see this when calling make test:

services/elasticache/pkg/resource/replication_group/sdk.go:66:38: cannot use resp.ReplicationGroup.AuthTokenLastModifiedDate (type *time.Time) as type *"k8s.io/apimachinery/pkg/apis/meta/v1".Time in assignment

I have seen this in other generated service controllers as well. Basically, we need to "unpack" the wrapped time.Time that is within the metav1.Time struct:

https://github.com/kubernetes/kubernetes/blob/1ed2cf189571bf0264822adf0cfe88007a30e934/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/time.go#L34

Obviously, when assigning SDK output Shape fields that are of type time.Time to CRD fields that are of type metav1.Time, we'll need to do a similar conversion the other way, to remedy build failures like this:

services/elasticache/pkg/resource/cache_cluster/sdk.go:67:38: cannot use resp.CacheCluster.AuthTokenLastModifiedDate (type *time.Time) as type *"k8s.io/apimachinery/pkg/apis/meta/v1".Time in assignment

Generate CRDs from build-controller.sh script.

Generate CRD from build-controller.sh script.

Secrets Integration to Protect CRD Fields

The AWS RDS API, among others, requires sensitive fields to be displayed in plain text. To protect such fields, we want to:

• Allow users to create Kubernetes Secrets and store their sensitive data in them
• Receive Secret references in place of target fields and pass the Secret data to the desired AWS API

Link to the Secrets Integration design proposal

Testing

This design issue is about the ACK testing setup. Usage of requirements key words as per RFC 2119.

Scope

One of our ACK tenets is that it is optimized for production usage. This requires that we have a set of test suites in place. In scope for this issue is the selection of the nature of the tests carried out as well as the design of the test infrastructure.

Input

Suggested tests include (but are not limited to):

• Unit-level tests in Go.
• Per-AWS-service test: checks to what extent an ACK for a certain service covers the operations supported by said service.
• Performance and scale tests: to figure out the ACK behaviour under short-term load.
• Soak tests: to figure out the ACK behaviour under long-term load.

Requirements

1. Tests in ACK MUST be automated. All results SHOULD be surfaced as bot messages on GitHub issues.
2. Every ACK feature graduating to GA MUST have full test coverage.
3. The selection of tests, documentation thereof, as well as the test infrastructure MUST be completed before implementation begins.

EKS Service Controller

New ACK Service Controller

Support for Amazon EKS

List of API resources

List the API resources in order of importance to you:

• Cluster
• Nodegroup
• FargateProfile

failure to generate API types for RDS

Getting failure when generating the API types for RDS:

jaypipes@thelio:~/go/src/github.com/aws/aws-controllers-k8s$ ./scripts/build-controller.sh rds
Building Kubernetes API objects for rds
Generating deepcopy code for rds
panic: interface conversion: types.Type is nil, not *types.Named

goroutine 1 [running]:
sigs.k8s.io/controller-tools/pkg/deepcopy.shouldBeCopied(0xc00029d7c0, 0xc0048a18c0, 0x403901)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/deepcopy/traverse.go:611 +0x35a
sigs.k8s.io/controller-tools/pkg/deepcopy.(*ObjectGenCtx).GenerateForPackage.func2(0xc0048a18c0)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/deepcopy/gen.go:226 +0xac
sigs.k8s.io/controller-tools/pkg/markers.EachType.func1(0xc000796500, 0xc0006ae700, 0xc0006be2a0)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/markers/zip.go:179 +0x779
sigs.k8s.io/controller-tools/pkg/loader.(*typeVisitor).Visit(0xc0047b8b80, 0xdf8160, 0xc0006be2a0, 0x0, 0x0)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/loader/visit.go:64 +0x140
go/ast.Walk(0xdf2780, 0xc0047b8b80, 0xdf8160, 0xc0006be2a0)
	/usr/local/go/src/go/ast/walk.go:52 +0x66
go/ast.Walk(0xdf2780, 0xc0047b8b80, 0xdf7ba0, 0xc0006ae700)
	/usr/local/go/src/go/ast/walk.go:331 +0xf3b
go/ast.walkDeclList(0xdf2780, 0xc0047b8b80, 0xc0007b4200, 0x18, 0x20)
	/usr/local/go/src/go/ast/walk.go:38 +0x9e
go/ast.Walk(0xdf2780, 0xc0047b8b80, 0xdf7a60, 0xc000796500)
	/usr/local/go/src/go/ast/walk.go:353 +0x264e
sigs.k8s.io/controller-tools/pkg/loader.EachType(0xc00029d7c0, 0xc0047b8b60)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/loader/visit.go:38 +0xaa
sigs.k8s.io/controller-tools/pkg/markers.EachType(0xc00029d7e0, 0xc00029d7c0, 0xc002ee0120, 0x8, 0xc0047cddc8)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/markers/zip.go:155 +0xa3
sigs.k8s.io/controller-tools/pkg/deepcopy.(*ObjectGenCtx).GenerateForPackage(0xc000af9b40, 0xc00029d7c0, 0xcd2acf, 0x5, 0xcd1e08)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/deepcopy/gen.go:216 +0x1da
sigs.k8s.io/controller-tools/pkg/deepcopy.Generator.Generate(0xc00009862b, 0x5d, 0x0, 0x0, 0xc0000a6140, 0xc0001ce701, 0xc0000a6140)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/deepcopy/gen.go:147 +0x1a5
sigs.k8s.io/controller-tools/pkg/genall.(*Runtime).Run(0xc00017d680, 0xc0001fa340)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/genall/genall.go:171 +0x15e
main.main.func1(0xc0001eb8c0, 0xc0001fa340, 0x2, 0x2, 0x0, 0x0)
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/cmd/controller-gen/main.go:176 +0xa6
github.com/spf13/cobra.(*Command).execute(0xc0001eb8c0, 0xc000032190, 0x2, 0x2, 0xc0001eb8c0, 0xc000032190)
	/home/jaypipes/go/pkg/mod/github.com/spf13/[email protected]/command.go:838 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001eb8c0, 0xc0001fa2e0, 0x4, 0x0)
	/home/jaypipes/go/pkg/mod/github.com/spf13/[email protected]/command.go:943 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
	/home/jaypipes/go/pkg/mod/github.com/spf13/[email protected]/command.go:883
main.main()
	/home/jaypipes/go/pkg/mod/sigs.k8s.io/[email protected]/cmd/controller-gen/main.go:200 +0x34a
Building service controller for rds

I'm going to look into this and see what the deal is with the types.Named type.

Unable to generate apigatewayv2 API (deepcopy-gen failure)

When attempting to generate the service controller for the APIGatewayV2 API, we're getting the following error:

panic: interface conversion: types.Type is nil, not *types.Named

goroutine 1 [running]:
sigs.k8s.io/controller-tools/pkg/deepcopy.shouldBeCopied(0xc000111120, 0xc0033f4180, 0x403901)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/deepcopy/traverse.go:613 +0x451
sigs.k8s.io/controller-tools/pkg/deepcopy.(*ObjectGenCtx).GenerateForPackage.func2(0xc0033f4180)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/deepcopy/gen.go:226 +0xac
sigs.k8s.io/controller-tools/pkg/markers.EachType.func1(0xc000882380, 0xc000728500, 0xc00078e060)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/markers/zip.go:179 +0x779
sigs.k8s.io/controller-tools/pkg/loader.(*typeVisitor).Visit(0xc00315b2c0, 0xdf8000, 0xc00078e060, 0x0, 0x0)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/loader/visit.go:64 +0x140
go/ast.Walk(0xdf2620, 0xc00315b2c0, 0xdf8000, 0xc00078e060)
	/usr/local/go/src/go/ast/walk.go:52 +0x66
go/ast.Walk(0xdf2620, 0xc00315b2c0, 0xdf7a40, 0xc000728500)
	/usr/local/go/src/go/ast/walk.go:331 +0xf3b
go/ast.walkDeclList(0xdf2620, 0xc00315b2c0, 0xc0000cb100, 0xc, 0x10)
	/usr/local/go/src/go/ast/walk.go:38 +0x9e
go/ast.Walk(0xdf2620, 0xc00315b2c0, 0xdf7900, 0xc000882380)
	/usr/local/go/src/go/ast/walk.go:353 +0x264e
sigs.k8s.io/controller-tools/pkg/loader.EachType(0xc000111120, 0xc00315b1c0)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/loader/visit.go:38 +0xaa
sigs.k8s.io/controller-tools/pkg/markers.EachType(0xc000111140, 0xc000111120, 0xc001ce8e70, 0x8, 0xc00249db88)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/markers/zip.go:155 +0xa3
sigs.k8s.io/controller-tools/pkg/deepcopy.(*ObjectGenCtx).GenerateForPackage(0xc002d27b40, 0xc000111120, 0xcd2b2f, 0x5, 0xcd1e68)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/deepcopy/gen.go:216 +0x1da
sigs.k8s.io/controller-tools/pkg/deepcopy.Generator.Generate(0x0, 0x0, 0x0, 0x0, 0xc0001120a0, 0xc0001d6701, 0xc0001120a0)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/deepcopy/gen.go:147 +0x1a5
sigs.k8s.io/controller-tools/pkg/genall.(*Runtime).Run(0xc000044680, 0xc000202340)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/pkg/genall/genall.go:171 +0x15e
main.main.func1(0xc0001a5b80, 0xc000202340, 0x2, 0x2, 0x0, 0x0)
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go:176 +0xa6
github.com/spf13/cobra.(*Command).execute(0xc0001a5b80, 0xc000032190, 0x2, 0x2, 0xc0001a5b80, 0xc000032190)
	/home/jaypipes/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001a5b80, 0xc0002022e0, 0x4, 0x0)
	/home/jaypipes/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/home/jaypipes/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
	/home/jaypipes/go/src/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go:200 +0x34a
exit status 2

Since controller-tools' deepcopy-gen has virtually no debugging capability, I had to hack a local git clone of controller-tools up and re-ran the deepcopy-gen command with my local version to see what was going on:

jaypipes@thelio:~/go/src/github.com/aws/aws-controllers-k8s$ /home/jaypipes/go/src/sigs.k8s.io/controller-tools/.run-controller-gen.sh object paths=./services/apigatewayv2/apis/v1alpha1/...
XXX: Processing APISpec
XXX: Processing APIStatus
XXX: Processing API
XXX: Processing APIList
XXX: Processing APIMappingSpec
XXX: Processing APIMappingStatus
XXX: Processing APIMapping
XXX: Processing APIMappingList
XXX: Processing AuthorizerSpec
XXX: Processing AuthorizerStatus
XXX: Processing Authorizer
XXX: Processing AuthorizerList
XXX: Processing DeploymentSpec
XXX: Processing DeploymentStatus
XXX: Processing Deployment
XXX: Processing DeploymentList
XXX: Processing DomainNameSpec
XXX: Processing DomainNameStatus
XXX: Processing DomainName
XXX: Processing DomainNameList
XXX: Processing AuthorizationType
XXX: Processing AuthorizerType
XXX: Processing ConnectionType
XXX: Processing ContentHandlingStrategy
XXX: Processing DeploymentStatus_SDK
XXX: Processing DomainNameStatus_SDK
XXX: Processing EndpointType
XXX: Processing IntegrationType
XXX: Processing LoggingLevel
XXX: Processing PassthroughBehavior
XXX: Processing ProtocolType
XXX: Processing SecurityPolicy
XXX: Processing VPCLinkStatus_SDK
XXX: Processing VPCLinkVersion
XXX: Processing IntegrationSpec
XXX: Processing IntegrationStatus
XXX: Processing Integration
XXX: Processing IntegrationList
XXX: Processing IntegrationResponseSpec
XXX: Processing IntegrationResponseStatus
XXX: Processing IntegrationResponse
XXX: Processing IntegrationResponseList
XXX: Processing ModelSpec
XXX: Processing ModelStatus
XXX: Processing Model
XXX: Processing ModelList
XXX: Processing RouteSpec
XXX: Processing RouteStatus
XXX: Processing Route
XXX: Processing RouteList
XXX: Processing RouteResponseSpec
XXX: Processing RouteResponseStatus
XXX: Processing RouteResponse
XXX: Processing RouteResponseList
XXX: Processing StageSpec
XXX: Processing StageStatus
XXX: Processing Stage
XXX: Processing StageList
XXX: Processing API
<ERROR OUTPUT>

Previous fix for #89 only fixed Enum types apparently and not TypeDef types that have conflicting names with CRDs. In the case of the APIGatewayV2 API, there is a Shape called API that is conflicting with the top-level resource name (the CRD's Kind) of API :(

Add usage examples

It would be great to have examples of how to create services using tools like cdk8s and Terraform kubernetes provider

Something that can be integrated into existing projects as dependencies and not raw kubernetes yaml files. 👍

Cannot deepcopy-gen `time.Time` fields

Example output when building RDS service controller after generating the API types:

# github.com/aws/aws-controllers-k8s/services/rds/apis/v1alpha1
services/rds/apis/v1alpha1/zz_generated.deepcopy.go:134:8: (*in).DeepCopyInto undefined (type *time.Time has no field or method DeepCopyInto)

Turns out we will need to replace all *time.Time fields with *metav1.Time fields.

https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/time.go

metav1.Time is basically time.Time that has been manually deepcopy-gen'd

IAM Dependency potential issues

When considering https://github.com/aws/amazon-eks-pod-identity-webhook as the setup to translate IAM roles to PODS and than considering the Operator functionality we might come to this "issue".
Scenario:

• I have a deployment A
• Deployment A requires for example S3 bucket B
• Deployment A requires IAM role C with policy to access S3 bucket B
• I create 3 Objects in my repo one for Deployment one for Bucket and one for IAM
• Now I deploy those resources using GitOps flow

Outcome:
Unknown order and unknown behaviour of the final deployment pod, I assume the Webhook inability to get IAM role for Pod if it still does not exist will not result in Pod not being created meaning I would need to hack my readiness probe to check for both IAM env var and S3 availability to start my Pod.

I might be wrong but it seems this would be the case, I know this might be K8s limitation but maybe do you have other reasonable suggestions to solve this?

Access control

This design issue is about the access control model of the ASO. We want to provide a flexible and out-of-the-box secure way for ASO to access AWS services and resources. Usage of requirements key words as per RFC 2119.

Scope

A client is a human user or another program that interacts with the ASO by creating CRDs that define operations on an AWS service:

  [client] --> [ASO] --> [AWS service]

In this context the AWS Identity and Access Management (IAM) access management is in scope and the primary focus of this design issue.

Further, the ASO depends on Kubernetes Role-Based Access Control (RBAC) which is also in scope for this design issue.

Input

Input from roadmap issue aws/containers-roadmap#456:

• On defining alternative permissions:
  • @dcherman in aws/containers-roadmap#456 (comment)
  • @LeoAdamek in aws/containers-roadmap#456 (comment)
• On IAM permissions boundaries:
  • @sftim in aws/containers-roadmap#456 (comment)
• On OPA integration:
  • @daviddyball in aws/containers-roadmap#456 (comment)

Requirements

1. The ASO access control model MUST follow the least privilege principle. For example, giving the ASO the permission to access all AWS services independent of the client would violate this requirement.
2. By default, the permissions ASO gets granted for an AWS service MUST be the permissions the requesting client has, concerning said AWS service.
3. ASO access control SHOULD support limiting or extending permissions beyond the requesting client's permissions.
4. ASO access control SHOULD support IAM permissions boundaries.
5. ASO access control SHOULD be able to integrate with Open Policy Agent (OPA).

Resources

• IAM roles for service accounts
• How does IAM and RBAC play together?
• RBAC in Kubebuilder

determine primary API namespace

We should decide what the primary API namespace for the CRDs exposed by AWS Service Operator.

The original ASO used the namespace service-operator.aws for its CRDs:

https://github.com/amazon-archives/aws-service-operator/blob/b4befd62322a57ac78aa39ea08771fc32912592a/configs/aws-service-operator.yaml#L37

Other Kubernetes controllers that deal with AWS resources use different namespaces:

• ALB Ingress controller v2 uses k8s.aws: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/v2/config/crds/ingress_v1alpha1_loadbalancer.yaml#L7
• AppMesh controller for k8s uses k8s.aws: https://github.com/aws/aws-app-mesh-controller-for-k8s/blob/dbfce09bf701846fb7cacd8e57c424191334505b/deploy/all.yaml#L7
• ENIConfig from CNI plugin v1 uses k8s.amazonaws.com: https://github.com/aws/amazon-vpc-cni-k8s/blob/73d031441c8b20e73675fc26cd16bf016702152a/pkg/eniconfig/eniconfig.go#L152
• Sagemaker controller for k8s used aws.amazon.com: https://github.com/aws/amazon-sagemaker-operator-for-k8s/blob/1e4bee1f0b172ab83385d1aa113b22f065baf1c7/config/crd/bases/sagemaker.aws.amazon.com_models.yaml#L16

We should standardize on one namespace.

Better error logging for case of Cloud Formation errors.

Describe the bug
When I get k8s error log the cloud formation error seems to be truncated

Steps to reproduce
Cause an error and match the error displayed in k8s log with what you see in aws console for CloudFormation.

Expected outcome
The two logs should have the same information.

Environment
k8s 1.17

Create end-user docs

We need end-user documentation for the installation and usage of the software.

Identifying AWS resources created from ACK from perspective of service team.

Is your feature request related to a problem?
API Gateway leadership wish to measure the success of onboarding with ACK by identifying the usage generated from API Gateway's onboarding with ACK.

Describe the solution you'd like
Up for discussion but maybe a tag that specifies when an aws resource is created from ACK controller.

Describe alternatives you've considered
None.

Create a dedicated channel on Kubernetes community Slack

Following the steps outlined in the Kubernetes Slack guide create a channel #aws-service-operator.

Write integration tests for apigatewayv2

• Write integration tests for apigatewayv2 after manual testing.

Create test infra

As per #6 create a test infrastructure

Difference between service broker

I'd like help understanding the goals of aws-service-operator compared to the existing service broker. Is it intended that both products will exist and overlap in functionality or will there be features/resources expected that will only be possible in one vs the other?

Looking at the existing/old implementation for the service operator I can see what AWS services are targeted but it would be nice to have clarity for customers on when they should use one vs another.