newrelic / k8s-webhook-cert-manager

Generate certificate suitable for use with any Kubernetes Mutating Webhook.

Kubernetes Webhook Certificate Manager

Script to generate a certificate suitable for use with any Kubernetes Mutating or Validating Webhook.

To be able to execute the script in a Kubernetes cluster, it's released as a Docker image and can be executed, for instance, as a Kubernetes Job.

This is a detailed list of steps the script is executing:

• Generate a server key.
• If there is any previous CSR (certificate signing request) for this key, it is deleted.
• Generate a CSR for such key.
• The signature of the key is then approved.
• The server's certificate is fetched from the CSR and then encoded.
• A secret of type tls is created with the server certificate and key.
• The k8s extension api server's CA bundle is fetched.
• The mutating webhook configuration for the webhook server is patched with the k8s api server's CA bundle from the previous step. This CA bundle will be used by the k8s extension api server when calling our webhook.

If you wish to learn more about TLS certificates management inside Kubernetes, check out the official documentation for Managing TLS Certificate in a Cluster.

Usage example

The script expects multiple mandatory arguments. This is an example:

./generate_certificate.sh --service ${WEBHOOK_SERVICE_NAME} --webhook
${WEBHOOK_NAME} --secret ${SECRET_NAME} --namespace ${WEBHOOK_NAMESPACE} 

Development setup

This script is designed to run in Kubernetes clusters. For development purposes, we recommend using Minikube.

Support

Should you need assistance with New Relic products, you are in good hands with several support diagnostic tools and support channels.

This troubleshooting framework steps you through common troubleshooting questions.

New Relic offers NRDiag, a client-side diagnostic utility that automatically detects common problems with New Relic agents. If NRDiag detects a problem, it suggests troubleshooting steps. NRDiag can also automatically attach troubleshooting data to a New Relic Support ticket. Remove this section if it doesn't apply.

If the issue has been confirmed as a bug or is a feature request, file a GitHub issue.

Support Channels

• New Relic Documentation: Comprehensive guidance for using our platform
• New Relic Community: The best place to engage in troubleshooting questions
• New Relic Developer: Resources for building a custom observability applications
• New Relic University: A range of online training for New Relic users of every level
• New Relic Technical Support 24/7/365 ticketed support. Read more about our Technical Support Offerings.

Privacy

At New Relic we take your privacy and the security of your information seriously, and are committed to protecting your information. We must emphasize the importance of not sharing personal data in public forums, and ask all users to scrub logs and diagnostic information for sensitive information, whether personal, proprietary, or otherwise.

We define “Personal Data” as any information relating to an identified or identifiable individual, including, for example, your name, phone number, post code or zip code, Device ID, IP address, and email address.

For more information, review New Relic’s General Data Privacy Notice.

Contribute

We encourage your contributions to improve this project! Keep in mind that when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project.

If you have any questions, or to execute our corporate CLA (which is required if your contribution is on behalf of a company), drop us an email at [email protected].

A note about vulnerabilities

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

If you would like to contribute to this project, review these guidelines.

To all contributors, we thank you! Without your contribution, this project would not be what it is today.

License

This project is licensed under the Apache 2.0 License.

k8s-webhook-cert-manager's Issues

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error	❌ Fail	⚠️ Warn	✅ Pass	Ignored	Total
0	3	1	3	0	7

• Fail
  • ❌ readme-starts-with-community-plus-header
  • ❌ readme-contains-link-to-security-policy
  • ❌ code-of-conduct-should-not-exist-here
• Warning
  • ⚠️ third-party-notices-file-exists
• Passed
  • ✅ license-file-exists
  • ✅ readme-file-exists
  • ✅ readme-contains-discuss-topic

Fail #

❌ readme-starts-with-community-plus-header #

The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/. Below is a list of files or patterns that failed:

• README.md: The first 5 lines do not contain the pattern(s): Open source Community Plus header (see https://opensource.newrelic.com/oss-category).
  • 🔨 Suggested Fix: prepend [![Community Plus header](https://github.com/newrelic/opensource-website/raw/master/src/images/categories/Community_Plus.png)](https://opensource.newrelic.com/oss-category/#community-plus) to file

❌ readme-contains-link-to-security-policy #

Doesn't contain a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

❌ code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Found files. Below is a list of files or patterns that failed:

• CODE_OF_CONDUCT.md
  • 🔨 Suggested Fix: Remove file

Warning #

Click to see rules

⚠️ third-party-notices-file-exists #

A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. Below is a list of files or patterns that failed:

• THIRD_PARTY_NOTICES*
• THIRD-PARTY-NOTICES*
• THIRDPARTYNOTICES*

Passed #

Click to see rules

✅ license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

✅ readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

✅ readme-contains-discuss-topic #

Contains a link to the appropriate discuss.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate discuss.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

No kind "CertificateSigningRequest" is registered for version "certificates.k8s.io/v1" error

Description

When running this image in a 1.19 KIND cluster the kubectl get csr step fails because the version of kubectl is too old.

Expected Behavior

The docker images should work in 1.19

Troubleshooting or NR Diag results

Steps to Reproduce

Running version 1.3.0 of the docker image in a KIND 1.19 cluster.

Your Environment

KIND 1.19

Additional context

Updating to a newer version of kubectl that supporter certificates.k8s.io/v1 or fully qualifying the kubectl get should fix this issue. kubectl get certificatesigningrequests.v1beta1.certificates.k8s.io

Move k8s-webhook-cert-manager-release Job and travis tests

https://fsi-build.pdx.vm.datanerd.us/view/k8s-metadata/job/k8s-webhook-cert-manager-r
elease/

Notice that:

• The release in the job options directly in jenkins
• The tests are performed in a minikube spin up in Travis

We should make sure we follow a common pattern:
PR/Merge -> Run tests/build using a makefile
Create Tag -> Push release

newrelic-bundle-nri-metadata-injection-job failed

Description

newrelic-bundle-nri-metadata-injection-job failed to locate certificate approved.

Expected Behavior

Certificate approved.
./generate_certificate.sh script able to find approved certificate

Troubleshooting or NR Diag results

container log:

creating certs in tmpdir /tmp/tmp.cJcAIl
Generating RSA private key, 2048 bit long modulus (2 primes)
................................................................................................................................................................................................+++++
..........+++++
e is 65537 (0x010001)
creating csr: newrelic-bundle-nri-metadata-injection.master
Error from server (NotFound): certificatesigningrequests.certificates.k8s.io "newrelic-bundle-nri-metadata-injection.master" not found
certificatesigningrequest.certificates.k8s.io/newrelic-bundle-nri-metadata-injection.master created
NAME AGE SIGNERNAME REQUESTOR CONDITION
newrelic-bundle-nri-metadata-injection.master 0s kubernetes.io/legacy-unknown system:serviceaccount:master:newrelic-bundle-nri-metadata-injection Pending
certificatesigningrequest.certificates.k8s.io/newrelic-bundle-nri-metadata-injection.master approved
ERROR: After approving csr newrelic-bundle-nri-metadata-injection.master, the signed certificate did not appear on the resource. Giving up after 1 minute.

kubectl command

kubectl get certificatesigningrequests.v1beta1.certificates.k8s.io/nri-bundle-nri-metadata-injection.master
NAME AGE SIGNERNAME REQUESTOR CONDITION
nri-bundle-nri-metadata-injection.master 5m50s kubernetes.io/legacy-unknown u-34muy35xxv Approved

kubectl command

kubectl get certificatesigningrequests.v1beta1.certificates.k8s.io/nri-bundle-nri-metadata-injection.master -o json

{
"apiVersion": "certificates.k8s.io/v1beta1",
"kind": "CertificateSigningRequest",
"metadata": {
"creationTimestamp": "2021-02-12T12:26:03Z"

 deleted  

}
"status": {
    "conditions": [
        {
            "lastTransitionTime": "2021-02-12T12:26:03Z",
            "lastUpdateTime": "2021-02-12T12:26:03Z",
            "message": "This CSR was approved by kubectl certificate approve.",
            "reason": "KubectlApprove",
            "status": "True",
            "type": "Approved"
        }
    ]
}

}

script failed to find approved certificate

verify certificate has been signed
i=1
while [ "$i" -ne 20 ]
do
serverCert=$(kubectl get "${csr}/${csrName}" -o jsonpath='{.status.certificate}')
if [ "${serverCert}" != '' ]; then
break
fi
sleep 10
i=$((i + 1))
done

kubectl command

kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
xxxx1 Ready controlplane,etcd,worker 3h48m v1.19.7 10.97.13.191 Red Hat Enterprise Linux Server 7.9 (Maipo) 3.10.0-1160.15.2.el7.x86_64 docker://1.13.1

kubectl command

kubectl get certificatesigningrequests.v1beta1.certificates.k8s.io/nri-bundle-nri-metadata-injection.master
NAME AGE SIGNERNAME REQUESTOR CONDITION
nri-bundle-nri-metadata-injection.master 37m kubernetes.io/legacy-unknown u-34muy35xxv Approved

kubectl command

kubectl get certificatesigningrequests.v1beta1.certificates.k8s.io/nri-bundle-nri-metadata-injection.master -o 'jsonpath={.status.certificate}'

kubectl get certificatesigningrequests.v1beta1.certificates.k8s.io/nri-bundle-nri-metadata-injection.master -o 'jsonpath={.status.conditions[0].status}'
True

Steps to Reproduce

helm repo add newrelic https://helm-charts.newrelic.com

helm install newrelic-bundle newrelic/nri-bundle
--set global.licenseKey=LICENCE_KEY
--set global.cluster=mycluster
--namespace=master
--set newrelic-infrastructure.privileged=true
--set ksm.enabled=true
--set kubeEvents.enabled=true
--set logging.enabled=true

Your Environment

Rancher 2.5.5
v1.19.7
Red Hat Enterprise Linux Server 7.9
docker 1.13.1

Additional context

Archive repository

When newrelic/helm-charts#334 gets merged, this repo will no longer be used and will be marked read-only.

No bugfixes or other support will be offered for this software.

/run/secrets/kubernetes.io/serviceaccount/ca.crt: No such file or directory

I run the cert gen script on my laptop (Mac OS) and got the following errors:

creating certs in tmpdir /var/folders/6b/wmx7z7bs7sx3r7tx_rt9z6cc0000gp/T/tmp.Y6seDAGr 
Generating RSA private key, 2048 bit long modulus
.........+++
..................................+++
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io "k8sbot-webhook-service.k8sbot-system" deleted
WARN: Previous CSR was found and removed.
certificatesigningrequest.certificates.k8s.io/k8sbot-webhook-service.k8sbot-system created
NAME                                   AGE   SIGNERNAME                     REQUESTOR          CONDITION
k8sbot-webhook-service.k8sbot-system   0s    kubernetes.io/legacy-unknown   kubernetes-admin   Pending
certificatesigningrequest.certificates.k8s.io/k8sbot-webhook-service.k8sbot-system approved
secret/k8sbot-webhook-server-cert configured
./webhook_cert_gen.sh: line 166: /run/secrets/kubernetes.io/serviceaccount/ca.crt: No such file or directory
INFO: Trying to patch webhook adding the caBundle.
mutatingwebhookconfiguration.admissionregistration.k8s.io/k8sbot-mutating-webhook-configuration patched (no change)

Certificate in openshift 4.x

Description

When deploying on openshift 4.x the metadata injector doesn't work as expected.
The root cause of the issue seems to be the certificate generation.

We are seeing checking the logs of the nri-bundle-nri-metadata-injection:

2020/10/16 09:45:12 http: TLS handshake error from 10.131.0.29:37428: remote error: tls: unknown certificate authority
2020/10/16 09:53:39 http: TLS handshake error from 10.129.0.1:49314: remote error: tls: bad certificate

We checked the certificate with the CA provided by MutatingWebhookConfiguration and in a random minikube is valid in openshift it is not.

That CA is added by our script, that takes it from /run/secrets/kubernetes.io/serviceaccount/ca.crt, but the certificated generated and placed into nri-bundle-nri-metadata-injection is signed by a different CA

For older version of openshift (3.x) there is a Guide to follow in order to set it up properly, but it does not apply for 4.x

Expected Behavior

The Metadata injection should work properly

Steps to Reproduce

In an openshift 4.x cluster install nri-bundle