openshift / aws-account-operator

Operator to manage pool of AWS accounts for Hive

AWS Account Operator

General Overview

The aws-account-operator is responsible for creating and maintaining a pool of AWS accounts and assigning accounts to AccountClaims. The operator creates the account in AWS, does the initial setup and configuration of those accounts, creates IAM resources and exposes credentials for an IAM user with enough permissions to provision an OpenShift 4.x cluster.

The operator is deployed to an OpenShift cluster in the aws-account-operator namespace.

Documentation

For information on the inner-workings, installation, development and testing of the operator, please refer to our Documentation.

Boilerplate

This repository subscribes to the openshift/golang-osd-operator convention of boilerplate. See the README for details about the functionality that brings in.

aws-account-operator's Issues

2023 Change Freeze | branch:master

master branch is frozen for merges until Friday Jan-06-2023 @ 14:00 UTC

Future Release Branches Frozen For Merging | branch:release-4.7

The following branches are being fast-forwarded from the current development branch (release-4.6) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.7

Contact the Test Platform or Automated Release teams for more information.

Migrate aws-account-operator to Go 1.18

Right now the boilerplate ci checks only work, when running Go 1.17.

We should think about migrating to Go 1.18 - as this is the current major version (and also the default on Fedora 36).

As part of that, the boilerplate needs to be updated as well, as golangci-lint used by it currently is version 1.30.0, which is not compatible with Go 1.18:

ERRO Running error: buildir: failed to load package goarch: could not load export data: cannot import "internal/goarch" (unknown iexport format version 2), export data is newer version - update tool

Finalizer prevents namespace from being deleted

We cannot terminate namespaces that have accountclaim.aws.managed.openshift.io objects on them.

It seems we're have an issue with the finalizers

Finalizers are arbitrary string values, that when present ensure that a hard delete of a resource is not possible while they exist.

The first delete request on an object with finalizers sets a value for the metadata.deletionTimestamp field but does not delete it. Once this value is set, entries in the finalizer list can only be removed.

tl;dr if the finalizer does not succeed, the object cannot be deleted.
In this case we have an object in the namespaces that are to be terminated:

# kubectl get accountclaim.aws.managed.openshift.io/mycluster -o yaml
apiVersion: aws.managed.openshift.io/v1alpha1
kind: AccountClaim
metadata:
creationTimestamp: 2019-08-06T10:01:22Z
deletionGracePeriodSeconds: 0
deletionTimestamp: 2019-08-06T10:48:15Z
finalizers:
- finalizer.aws.managed.openshift.io 

once we remove the finalizer from this object, it can be removed and the namespace can be terminated.

This is also reflected in the logs:

{"level":"error","ts":1565089405.1570218,"logger":"controller_accountclaim","msg":"Failed to get claimed account","Request.Namespace":"REDACTED_FOR_ISSUE","Request.Name":"mycluster","error":"Account.aws.managed.openshift.io \"\" not found","stacktrace":"github.com/openshift/aws-account-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/openshift/aws-account-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/openshift/aws-account-operator/pkg/controller/accountclaim.(*ReconcileAccountClaim).Reconcile\n\tsrc/github.com/openshift/aws-account-operator/pkg/controller/accountclaim/accountclaim_controller.go:127\ngithub.com/openshift/aws-account-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsrc/github.com/openshift/aws-account-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/openshift/aws-account-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tsrc/github.com/openshift/aws-account-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tsrc/github.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsrc/github.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tsrc/github.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1565089405.1570847,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"accountclaim-controller","request":"REDACTED_FOR_ISSUEr","error":"Secret \"aws\" not found","stacktrace":"github.com/openshift/aws-account-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/openshift/aws-account-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/openshift/aws-account-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsrc/github.com/openshift/aws-account-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/openshift/aws-account-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tsrc/github.com/openshift/aws-account-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tsrc/github.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsrc/github.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tsrc/github.com/openshift/aws-account-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}"

Migrate fakeclient to envtest

If we update the golangci-lint from 1.30 to the latest version (1.39) it's going to fail with:

GOLANGCI_LINT_CACHE=/tmp/golangci-cache golangci-lint run -c boilerplate/openshift/golang-osd-operator/golangci.yml ./...
pkg/totalaccountwatcher/totalaccountwatcher_test.go:21:2: SA1019: package sigs.k8s.io/controller-runtime/pkg/client/fake is deprecated: please use pkg/envtest for testing. This package will be dropped before the v1.0.0 release. Package fake provides a fake client for testing. (staticcheck)
        fakekubeclient "sigs.k8s.io/controller-runtime/pkg/client/fake"
        ^
pkg/controller/accountpool/accountpool_controller_test.go:17:2: SA1019: package sigs.k8s.io/controller-runtime/pkg/client/fake is deprecated: please use pkg/envtest for testing. This package will be dropped before the v1.0.0 release. Package fake provides a fake client for testing. (staticcheck)
        "sigs.k8s.io/controller-runtime/pkg/client/fake"
        ^
pkg/controller/accountclaim/accountclaim_controller_test.go:20:2: SA1019: package sigs.k8s.io/controller-runtime/pkg/client/fake is deprecated: please use pkg/envtest for testing. This package will be dropped before the v1.0.0 release. Package fake provides a fake client for testing. (staticcheck)
        "sigs.k8s.io/controller-runtime/pkg/client/fake"
        ^
make: *** [go-check] Error 1

Changing the fake client with pkg/envtest will cause problems because the NewFakeClient has been dropped in the envtest pkg. See: kubernetes-sigs/controller-runtime#1101

How to test this:

Current version works: docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.30.0 golangci-lint run -v
Latest version fails: docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.39.0 golangci-lint run -v

Workaround

Since I am working on multiple Golang projects which they are using different version of golang-ci, one way to overcome this issue is doing this at your .bashrc or .zshenv: alias golangci-lint="docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.30.0 golangci-lint"

PoolSize should be an int

@fahlmant The CR for the accountpool specifies that the account PoolSize is a string when it should be an int

2019 Change Freeze | branch:master

Master branch is frozen for merges until Monday Jan-06-2019 @ 14:00 UTC

BYOC account claim is not deleted

Deleting the namespace doesn't delete the account claim, causing the namespace to be stuck in terminating mode forever.

$ oc get namespace uhc-igoihman-1a45c1mq4tjh13c1enj3lu690lmm2atp
NAME                                            STATUS        AGE
uhc-igoihman-1a45c1mq4tjh13c1enj3lu690lmm2atp   Terminating   9h

$ oc get accountclaim -n uhc-igoihman-1a45c1mq4tjh13c1enj3lu690lmm2atp
NAME            STATE     ACCOUNT                 AGE
igoihmanbyoc4   Ready     osd-creds-mgmt-mtfqlj   9h

2020 Change Freeze | branch:master

Master branch is frozen for merges until Monday Jan-11-2021 @ 14:00 UTC

Future Release Branches Frozen For Merging | branch:release-4.5 branch:release-4.4

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.5
• release-4.4

Contact the Test Platform or Automated Release teams for more information.

OLM Issue preventing rollout of new versions to stage | Merge Blocker

There's currently an OLM issue caused by upstream issues when parsing the CRD validation.

The issue is making OLM block the rollout of new releases of the aws-account-operator, based on CRD validation of AccountClaim CRs.

While we don't have the OLM issue fixed and rolled out to hive stage shards, we'll be blocking new merges to the master branch.