Web Hook in cert-manager is not working properly. Can anyone please me out. about cert-manager

[minigamkreddy]

Thanks,
For replying back.
The issue got resolved I am able to create the certificate.

But the Certificate which was created not able to access in different namespaces using clusterissuer as Kind.

Do you have any reference yaml file. So that I can create the certificate which can be accessible across namespaces .

[hawksight]

Hey @minigamkreddy thanks for raising. This is a fairly common error to see and usually it is networking, DNS or cloud provider specific as the issue. We have a guide to help debug if you could try that first?

https://cert-manager.io/docs/troubleshooting/webhook/

Failing that, can you please share your k8s environment details.

In general every cert-manager resource if sent to the cert-manager-webhook deployment to validate the resource before it is saved to k8s to be actioned. It appears k8s cannot find that service, so please check that component is running.

[minigamkreddy]

Thanks For replying back

Yes I will Follow the link which you have provided me.

CERT MANAGER DETAILS
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7ddd8cdb9f-c7kwx 1/1 Running 1 38h
cert-manager-cainjector-57cd76c845-fk77m 1/1 Running 1 38h
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 38h

Environment Details
Kubernetes version:
kubectl version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1

OS Details
Description: Ubuntu 20.04.6 LTS

cert-manager version: ert-manager-controller:
Image: quay.io/jetstack/cert-manager-controller:v1.14.5

Note : For more Details related to Environment please reply back.

MORE DEATAILS
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get endpoints -n cert-manager cert-manager-webhook
NAME ENDPOINTS AGE
cert-manager-webhook 192.168.224.104:10250 38h

kubectl get pod -n cert-manager -l app.kubernetes.io/name=webhook
NAME READY STATUS RESTARTS AGE
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 39h

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get pod -n cert-manager -l app.kubernetes.io/name=webhook
NAME READY STATUS RESTARTS AGE
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 39h

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl logs -n cert-manager -l app.kubernetes.io/name=webhook | head -10
W0508 09:55:18.187126 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0508 09:55:18.273616 1 webhook.go:129] "using dynamic certificate generating using CA stored in Secret resource" logger="cert-manager.webhook" secret_namespace="cert-manager" secret_name="cert-manager-webhook-ca"
I0508 09:55:18.273781 1 server.go:146] "listening for insecure healthz connections" logger="cert-manager" address=":6080"
I0508 09:55:18.274173 1 server.go:206] "listening for secure connections" logger="cert-manager" address=":10250"
I0508 09:55:18.303989 1 reflector.go:351] Caches populated for *v1.Secret from k8s.io/[email protected]/tools/cache/reflector.go:229
I0508 09:55:19.280437 1 dynamic_source.go:255] "Updated cert-manager TLS certificate" logger="cert-manager" DNSNames=["cert-manager-webhook","cert-manager-webhook.cert-manager","cert-manager-webhook.cert-manager.svc"]
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms#

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get deploy -n cert-manager cert-manager-webhook -oyaml | grep -A3 ports:
ports:
- containerPort: 10250
name: https
protocol: TCP

[minigamkreddy]

resource mapping not found for name: "cert-manager" namespace: "" from "test-resources.yaml": no matches for kind "Issuer" in version "v1"
ensure CRDs are installed first
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxyconnect tcp: dial tcp 10.10.224.60:3128: connect: connection refused
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxyconnect tcp: dial tcp 10.10.224.60:3128: connect: connection
refused
When I do the kubectl apply -f test-resources.yaml cert-manager is excepted the response from the proxy server.
10.10.224.60:3128: these should not happen cert-manager should except the response from the current cluster.

Below commands are not working
kubectl -n cert-manager port-forward deploy/cert-manager-webhook 6080

curl -sS --dump-header - 127.0.0.1:6080/healthz => These command except the response from there 10.10.224.60:3128. These is not communicated with interval cluster.

How to reslove these issue.

[hawksight]

If you run kubectl get crd do you see the cert-manager CRD resources in the cluster?
Just looking at this error:

resource mapping not found for name: "cert-manager" namespace: "" from "test-resources.yaml": no matches for kind "Issuer" in version "v1"
ensure CRDs are installed first

Perhaps the CRDs were not installed, and that could explain it. But i expect they are there otherwise there should be more failing components.

[minigamkreddy]

root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl get crd
NAME CREATED AT
bgpconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
bgppeers.crd.projectcalico.org 2023-09-05T04:25:42Z
blockaffinities.crd.projectcalico.org 2023-09-05T04:25:42Z
caliconodestatuses.crd.projectcalico.org 2023-09-05T04:25:42Z
certificaterequests.cert-manager.io 2024-05-07T14:58:36Z
certificates.cert-manager.io 2024-05-07T14:58:36Z
challenges.acme.cert-manager.io 2024-05-07T14:58:36Z
clusterinformations.crd.projectcalico.org 2023-09-05T04:25:42Z
clusterissuers.cert-manager.io 2024-05-07T14:58:36Z
felixconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
globalnetworkpolicies.crd.projectcalico.org 2023-09-05T04:25:42Z
globalnetworksets.crd.projectcalico.org 2023-09-05T04:25:42Z
hostendpoints.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamblocks.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamconfigs.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamhandles.crd.projectcalico.org 2023-09-05T04:25:42Z
ippools.crd.projectcalico.org 2023-09-05T04:25:42Z
ipreservations.crd.projectcalico.org 2023-09-05T04:25:42Z
issuers.cert-manager.io 2024-05-07T14:58:36Z
kubecontrollersconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
networkpolicies.crd.projectcalico.org 2023-09-05T04:25:42Z
networksets.crd.projectcalico.org 2023-09-05T04:25:42Z
orders.acme.cert-manager.io 2024-05-07T14:58:36Z
root@KmasterVM:/home/manoj/VM1_E810/cert-manager#

root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
namespace/cert-manager unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable

[minigamkreddy]

above issue got resloved. But I am seeing another issue

oot@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
namespace/cert-manager-test unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z

I have change the time to IST but still it doesn't work

Can you suggest me any idea how to reslove these issue.

[hawksight]

Ok so you have CRDs and you now have a new error. Previously it was connection refused but now you have "tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z"

Can you get the cert-manager-webhook-ca certificate out of the secret it is stored in?

k get secret -n cert-manager cert-manager-webhook-ca -o json | jq -r '.data["tls.crt"]' | base64 --decode | openssl x509 -text -noout

Assuming that cert is active (which it should be). I would validate that your server time is synced properly. This seems like your master node you are running on. Maybe try checking your server time compared to actual time to see if there is any drift?

sntp -d pool.ntp.org

For example my output seems to be:

..
+0.022888 +/- 0.025390 pool.ntp.org 162.159.200.1

Which seems very much in sync.

The concerning bit in your error is that the cert seems to be issued for the future in your case. So some time setting appears to be off: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z

[hawksight]

The ClusterIssuer is available to use by Certificate resources from any namespace.
However the Certificate resource itself is namespaced and cert-manager will only create a Secret in the same namespace as the Certificate.

It very much depends on your use case here. Usually the certificates would be mounted to an app, provided to and Ingress in the same namespace and other controllers would reference the secret across namesapces, like an ingress controller deployment.

Can you describe the use case?
Do you know what resolved the original issue? Just incase anyone else has a similar problem in the future?

[minigamkreddy]

Hi @hawksight ,
Thanks for your response.

The original issue got resolved by fixing the proxy.
In our K8s, in /etc/kubernetes/ there were few files in which there was proxy set.
On removing the proxy, the issue got resolved.

Following is our usecase:

We have 2 applications running in Node1 and Node2 of single cluster.
APP1 running in Node1 in namespace node1
APP2 running in Node2 in namespace node2

We have common ClusterIssuer.yaml(currently we are using selfsigned) which should issue the certificate.
And the APP1 and APP2 helm charts will have respective certificate.yaml files which will refer to same ClusterIssuer.
We are able to create secret and mount it in APP1 and APP2.
We can see the ca.crt, tls.crt and tls.key in the mounted path in APP1 and APP2.

Now while doing certificate based authentication between APP1 and APP2, we are facing issue "root ca do not match".
We see that the ca.crt generated is different.
If we are referring same ClusterIssuer then the ca.crt should be same right ?

##############################################
Following is out ClusterIssue.yaml file:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-clusterissuer
namespace: test
spec:
selfSigned: {}

#####################################################
And following is our certificate in APP1 charts:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-ca
namespace: node1
spec:
isCA: true
commonName: "example1.com"
dnsNames:

• example1.com
  secretName: test-ca-secret
  issuerRef:
  name: selfsigned-clusterissuer
  kind: ClusterIssuer
  group: cert-manager.io

############################################
And following is our certificate file in APP2 charts:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-ca
namespace: {{ .Values.nameSpace }}
spec:
isCA: true
commonName: "example1.com"
dnsNames:

• example1.com
  secretName: test-ca-secret
  issuerRef:
  name: selfsigned-clusterissuer
  kind: ClusterIssuer
  group: cert-manager.io

Can you please check why the ca.crt getting generated are different ?

[ThatsMrTalbot]

The self signed issuer will create self signed certificates, so the two certificates created will not have the same root CA as they are self signed.

For your use case I would set up a CA issuer (which loads the CA from a secret) and use that. This would ensure certificates in both your apps have the same CA.

/priority awaiting-more-evidence