skyscanner / kms-issuer

KMS issuer is a cert-manager Certificate Request controller that uses AWS KMS to sign the certificate request.

KMS Issuer

KMS issuer is a cert-manager Certificate Request controller that uses AWS KMS to sign the certificate request.

Getting started

In this guide, we assume that you have a Kubernetes environment with a cert-manager version supporting CertificateRequest issuers, cert-manager v0.11.0 or higher.

For any details on Cert-Manager, check the official documentation.

Install

You can install the controller using the official helm chart:

helm repo add kms-issuer 'https://skyscanner.github.io/kms-issuer'
helm repo update

To install the chart with the release name kms-issuer:

helm upgrade --install kms-issuer kms-issuer/kms-issuer --namespace kms-issuer-system --create-namespace

Usage

1. Install cert-manager. The operator has been tested with version v0.15.1

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.yaml

2. Install and run the kms-issuer

Install the kms-issuer Kubernetes Custom Resources and start the controller.

# Install CRD
make install
# Run the controller (you must have have a role able to create/access KMS keys)
make run

3. Create a KMS Key

You need a valid KMS asymetric key that as the ability to SIGN_VERIFY messages. Currently, Cloudformation does not support KMS SIGN_VERIFY keys. To simply the provisioning process, the kms-issuer operator provides a dedicated controller for provisioning a valid KMS key.

cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSKey
metadata:
  name: kmskey-sample
spec:
  aliasName: alias/kms-issuer-example
  description: a kms-issuer example kms key
  customerMasterKeySpec: RSA_2048
  tags:
    project: kms-issuer
  deletionPolicy: Delete
  deletionPendingWindowInDays: 7
EOF

4. Create a KMS issuer object

cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSIssuer
metadata:
  name: kms-issuer
  namespace: default
spec:
  keyId: alias/kms-issuer-example # The KMS key id or alias
  commonName: My Root CA # The common name for the root certificate
  duration: 87600h # 10 years
EOF

At this point, the operator geneates a public root certificate signed using the provided KMS key. You can inspect it with the following command:

kubectl get kmsissuer kms-issuer -o json | jq -r ".status.certificate" |  base64 --decode  | openssl x509 -noout -text

6. Finally, create a Certificate request that will be signed by our KMS issuer.

cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
  namespace: default
spec:
  # Secret names are always required.
  secretName: example-com-tls
  duration: 8760h # 1 year
  renewBefore: 360h # 15d
  subject:
    organizations:
      - skyscanner
  # The use of the common name field has been deprecated since 2000 and is
  # discouraged from being used.
  commonName: example.com
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - server auth
    - client auth
  # At least one of a DNS Name, URI, or IP address is required.
  dnsNames:
    - example.com
    - www.example.com
  uris:
    - spiffe://cluster.local/ns/sandbox/sa/example
  ipAddresses:
    - 192.168.0.5
  # Issuer references are always required.
  issuerRef:
    name: kms-issuer
    # We can reference ClusterIssuers by changing the kind here.
    # The default value is Issuer (i.e. a locally namespaced Issuer)
    kind: KMSIssuer
    # This is optional since cert-manager will default to this value however
    # if you are using an external issuer, change this to that issuer group.
    group: cert-manager.skyscanner.net
EOF

You now have a key pair signed by KMS

kubectl get secret example-com-tls

API Reference

KMSKey

A KMSKey resource is used to create an AWS KMS asymetric key compatible with the KMS issuer.

Field	Type	Description
apiVersion	string	cert-manager.skyscanner.net/v1alpha1
kind	string	KMSKey
metadata	object	Refer to the Kubernetes API documentation for metadata fields.
spec	object	Desired state of the KMSKey resource.
spec.aliasName	string	the alias name for the kms key. This value must begin with alias/ followed by a name, such as alias/ExampleAlias.
spec.description	string	Description for the key. (optional)
spec.customerMasterKeySpec	string	Determines the signing algorithms that the CMK supports. Only RSA_2048 is currently supported. (optional, default=RSA_2048)
spec.policy	string	The key policy to attach to the CMK. (optional)
spec.tags	object	A list of tags for the key. (optional)
spec.deletionPolicy	string	Policy to deletes the alias and key on object deletion. Either Retain or Delete. (optional, default=Retain).
spec.deletionPendingWindowInDays	int	Number of days before the KMS key gets deleted. If you include a value, it must be between 7 and 30, inclusive. If you do not include a value, it defaults to 30. (optional)

KMSIssuer

A KMSIssuer resource configures a new Cert-Manager external issuer.

Field	Type	Description
apiVersion	string	cert-manager.skyscanner.net/v1alpha1
kind	string	KMSIssuer
metadata	object	Refer to the Kubernetes API documentation for metadata fields.
spec	object	Desired state of the KMSIssuer resource.
spec.keyId	string	The unique identifier for the customer master key
spec.commonName	string	The common name to be used on the Certificate.
spec.duration	duration	Certificate default Duration. (optional, default=26280h aka 3 years)
spec.renewBefore	duration	The amount of time before the certificate’s notAfter time that the issuer will begin to attempt to renew the certificate. If this value is greater than the total duration of the certificate (i.e. notAfter - notBefore), it will be automatically renewed 2/3rds of the way through the certificate’s duration. The NotBefore field on the certificate is set to the current time rounded down by the renewal interval. For example, if the certificate is renewed every hour, the NotBefore field is set to the beggining of the hour. If the certificate is renewed every day, the NotBefore field is set to the beggining of the day. This allows the generation of consistent certificates regardless of when it has been generated during the renewal period, or recreate the same certificate after a backup/restore of your kubernetes cluster. For more details on the computation, check the time.Truncate function.

Disable Approval Check

The KMS Issuer will wait for CertificateRequests to have an approved condition set before signing. If using an older version of cert-manager (pre v1.3), you can disable this check by supplying the command line flag -enable-approved-check=false to the Issuer Deployment.

Contributing

Kms-Issuer is built using the Kubebuilder framework. See the official documentation to get started and check CONTRIBUTING.md for more details.

Security

Check SECURITY.md.

kms-issuer's Issues

Docs: contriburting to cert-manager's Istio integration documentation

Currently cert-manager is missing Istio integration section in their docs. We could contribute that part.

KMSKey doesn't validate the configuration on creating which leads to errors when trying to delete it

AWS only supports deletionPendingWindowInDays from 7 to 30 days. When creating a resource there is no problem. It only raises the issue when you want to delete the key.

Creation:

cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSKey
metadata:
  name: kmskey-example1
spec:
  aliasName: alias/k8s-certs-kmskey-example1
  description: a kms-issuer example kms key
  customerMasterKeySpec: RSA_2048
  tags:
    Project: k8s
  deletionPolicy: Delete
  deletionPendingWindowInDays: 1
EOF
kmskey.cert-manager.skyscanner.net/kmskey-example1 created

Logs:

 2020-08-07T08:44:29.300Z    ERROR    controllers.kmskey_controller    Failed to delete the KMS key    {"kmskey": "/kmskey-example1", "error": "ValidationException: PendingWindowInDays must be between 7 and 30\n\tstatus code: 400, request
 id: 16f988b6-51eb-46e7-8d7a-6dc92df5309f"}
 github.com/go-logr/zapr.(*zapLogger).Error
     /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
 github.com/Skyscanner/kms-issuer/controllers.(*KMSKeyReconciler).manageFailure
     /workspace/controllers/kmskey_controller.go:138
 github.com/Skyscanner/kms-issuer/controllers.(*KMSKeyReconciler).Reconcile
     /workspace/controllers/kmskey_controller.go:88
 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256
 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
 k8s.io/apimachinery/pkg/util/wait.BackoffUntil
     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
 k8s.io/apimachinery/pkg/util/wait.JitterUntil
     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
 k8s.io/apimachinery/pkg/util/wait.Until
     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

Mitigation:

Manually edit KMSKey deletionPendingWindowInDays value to correct one and kms-issuer will schedule deletion of the key in AWS

Cluster level KMS Issuer

It would be very useful to have Cluster level KMS Issuer (ClusterKMSIssuer?) not to duplicate KMSIssuer objects in each namespace

Verbose issue on Public Key failure

It would be great if we could get visibility on errors in signer.Public:
https://github.com/Skyscanner/kms-issuer/blob/6445b18/pkg/signer/kmssigner.go#L50-L62

My idea is to move it to New() and KMSSigner structure and use Public to expose private variable. I'll create a PR as soon as I test the change.

Add E2E key deletetion test

We should add a test for when we delete the KMSKey object. It should mark the key for deleting on AWS.

Docs: examples of least privileged IAM policies for key generation and signing

We should add least privileged IAM policies for key generation and signing for current examples.

Simplify giithub-actions workflows

Summary

We currently have a bunch of separate workflows that run on each commit of a pull request. These include running unit tests, testing helm charts, deploying the controller via kusomize to a kind cluster to e2e testing, deploying the controller via helm to test helm, etc.

There is just a single job per workflow, with repeated steps across workflows (such as building the go binary, docker image, etc.).

It would be nice if we could create a single workflow for PR testing which using the pipeline nature of github actions to share artifacts, etc. between jobs (for example the go modules and docker image). Spinning up a kind cluster might not be able to be shared between jobs, but perhaps steps could be combined for both helm and e2e tests?

Add support for kubebuilder Multi-Version and Multi-Group API

Summary

We should follow the tutorial and add support for multi version and multi group APIs.

See https://book.kubebuilder.io/multiversion-tutorial/tutorial.html
See https://book.kubebuilder.io/migration/multi-group.html

Add support for ComponentConfig

Summary

We should add support for kubebuilder component config. See the docs here: https://book.kubebuilder.io/component-config-tutorial/tutorial.html

Notes

• we should update the helm chart to create the config map
• update code to support loading the config

Missing docker image

The controller deployment references an image that is not pushed to the docker registry. How are others installing this in an automated fashion?

Support importing existing root CA

Summary

Would it be possible to provide a root CA that is provisioned elsewhere? I'm thinking of the following use case:

1. Generate a CSR inter-ca.csr for the intermediate CA inter-ca with the KMS key alias/inter-ca-key and custom subject information
2. Generate a cert for the intermediate CA inter-ca.crt with the root key that is offline (not on KMS - since it does not allow importing existing asymmetric keys)
3. Create a KMS issuer using the intermediate certificate inter-ca.crt and the KMS key alias/inter-ca-key

I'm imagining something like this:

---
apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSKey
metadata:
  name: inter-ca-key
spec:
  aliasName: alias/inter-ca-key
  description: a kms-issuer example kms key for inter-ca
  customerMasterKeySpec: RSA_2048
  tags:
    project: kms-issuer
  deletionPolicy: Delete
  deletionPendingWindowInDays: 7

---
apiVersion: cert-manager.io/v1
kind: KMSImportedCertificate
metadata:
  name: issuer-ca-imported-cert
  namespace: default
spec:
  duration: 8760h # 1 year
  # renewBefore: 360h # 15d
  subject:
    organizations:
      - skyscanner
  commonName: example.com
  isCA: true
  usages: [...]
---
apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSIssuer
metadata:
  name: inter-ca-issuer
  namespace: default
spec:
  keyId: alias/inter-ca-key # The KMS key id or alias
  bootstrapCertificateRef:
    name: inter-ca-imported-cert
    kind: KMSImportedCertificate
    group: cert-manager.skyscanner.net

KMSImportedCertificate will have the CSR generated by the controller and stored in a CertificateRequest object. And when the cert is generated offline, it can be stored in its status.

Alternatively, bootstrapCertificateRef could reference an existing Certificate resource which is issued by some other issuer.

The spec of KMSImportedCertificate would be almost like Certificate but without the privateKey and secretName, and dnsNames/uris/ipAddresses are not relevant for an intermediate CA.

Add a kustomize release action

We want to have a process to create a single installation file generated by kustomize.

Kms-issuer doesn't support editing KMSKey objects

When editing tags in deployed KMSKey object kms-issuer is not updating AWS KMS key (e.g. tags nor the alias).

Support go module versioning

We use some of the packages in this repository in a few codebases and would be great to keep up with the updates. As the previous releases were tagged as major updates we can't import them anymore.

Would you consider to version the module so we can import it? I'd be happy to take a stab and contribute.

Error while creating Certificate Authority certificate: "MissingRegion: could not find region configuration"

Hi,
i'm trying to create a CA for cert-manager by using AWS KMS key. I have followed the guide but when creating the KMIssuer i have this erro on the controller manager:

ERROR   controllers.kmsissuer_controller        Failed to generate the Certificate Authority Certificate      
        {"name": "kms-issuer", "namespace": "cert-manager", "error": "MissingRegion: could not find region configuration"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/Skyscanner/kms-issuer/controllers.(*KMSIssuerReconciler).manageFailure
        /workspace/controllers/kmsissuer_controller.go:207
github.com/Skyscanner/kms-issuer/controllers.(*KMSIssuerReconciler).Reconcile
        /workspace/controllers/kmsissuer_controller.go:99
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99

My yaml file is:

apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSIssuer
metadata:
  name: kms-issuer
  namespace: default
spec:
  keyId: XXXXXXXXXX # The KMS key id or alias
  commonName: LabCa # The common name for the root certificate
  duration: 87600h # 10 years`

Could you help me to solve the issue, please?

Thanks
Cristian

Add e2e testing

We should add e2e tests which:

• spin up a kind cluster
• spin up https://github.com/nsmithuk/local-kms
• create a KMS issuer and make sure everything works as expected.

Adding retries on AWS operations

All AWS operations need to have retries not brick when e.g. IAM Role doesn't have sufficient permissions or API is throttled.

Clusterrole cannot approve certificate request from kms-issuer

Wrong issue. Closed.

Missing arm64 docker image

Currently the Github actions workflow for this project only publishes amd64 docker image, would you consider publishing arm64 images as well?

Support for custom labels

We would like to add custom labels to our deployment of this. Would the project be open to support for extra custom labels?

Root Ca is always not Before: Aug 31 00:00:00 2013 GM

Hey,

we noticed that the ca.crt which is created by the Kms Issuer has always notBefore: Aug 31 00:00:00 2013 GM and with a duration of 10 years it is only valid until Aug 29 00:00:00 2023 GMT. It does not matter when we create the KMSIssuer Object it is always the same Validity.

We build the Image with the Dockerfile in the kms-issuer repo.

Our KMSIssuer Object looks like this:

apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSIssuer
metadata:
  name: kms-issuer-unique-cn-test
  namespace: default
spec:
  commonName: selfsigned
  duration: 87600h
  keyId: alias/kms-issuer
  renewBefore: 4380

i assumed that the time in the container is somehow wrong configured and tried to set the duration to 20 years (175200h) which should extend the Root Ca until 2033 but this result in:

        Validity
            Not Before: Mar  2 12:00:00 2008 GMT
            Not After : Feb 26 12:00:00 2028 GMT

With 10 years (87600h) it looks like this:

        Validity
            Not Before: Aug 31 00:00:00 2013 GMT
            Not After : Aug 29 00:00:00 2023 GMT