Comments (9)
Hi @GarbageYard, looking at the events of the CertificateRequest, the ca-issuer
doesn't exist yet. Either check that the issuer you are trying to use have the correct name and type, or head over to the docs for more information on how to setup an issuer.
The Certificate
resource you are creating there has nothing to do with the csi-driver. The webhook error is still correct, and enforces that you must define one of those SANs on the Certificate resource for it to be accepted.
from csi-driver.
+1
from csi-driver.
I'm also experiencing this issue trying to mount to a container. Some more details, if it helps:
- Controller image:
quay.io/jetstack/cert-manager-controller:v1.7.2
- CSI driver image:
quay.io/jetstack/cert-manager-csi-driver:v0.2.0
- I'm running on Rancher K3s with kubelet 1.22.7. I have a few services using cert-manager certificates that are working fine.
Certificate definition:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: registry-tls
spec:
secretName: registry-tls-secret
dnsNames:
- 'registry.redacted.net'
issuerRef:
name: letsencrypt-ca
kind: ClusterIssuer
Pod definition:
apiVersion: v1
kind: Pod
metadata:
name: docker-registry-pod
labels:
app: registry
spec:
containers:
- name: registry
image: registry:2
volumeMounts:
- name: repo-vol
mountPath: '/var/lib/registry'
- name: tls
mountPath: '/tls'
volumes:
- name: repo-vol
persistentVolumeClaim:
claimName: docker-registry-vol-claim
- name: tls
csi:
readOnly: yes
driver: csi.cert-manager.io
volumeAttributes:
csi.cert-manager.io/issuer-name: letsencrypt-ca
csi.cert-manager.io/dns-names: registry.redacted.net
Logs from csi driver are a bunch of the following:
E0506 00:00:52.022851 1 server.go:109] driver "msg"="failed processing request" "error"="timed out waiting for the condition" "request"={} "rpc_method"="/csi.v1.Node/NodePublishVolume"
I0506 00:01:24.122823 1 nodeserver.go:74] driver "msg"="Registered new volume with storage backend" "pod_name"="docker-registry-pod"
I0506 00:01:24.122885 1 nodeserver.go:84] driver "msg"="Volume registered for management" "pod_name"="docker-registry-pod"
I0506 00:01:25.124239 1 manager.go:504] manager "msg"="Triggering new issuance" "volume_id"="csi-d00aa30048a62abe239705e4427130b3f8b9bec2b54487c35153d9a5ca3db6c1"
I0506 00:01:25.124309 1 manager.go:243] manager "msg"="Processing issuance" "volume_id"="csi-d00aa30048a62abe239705e4427130b3f8b9bec2b54487c35153d9a5ca3db6c1"
I0506 00:01:25.137035 1 manager.go:369] manager "msg"="Deleted CertificateRequest resource" "volume_id"="csi-d00aa30048a62abe239705e4427130b3f8b9bec2b54487c35153d9a5ca3db6c1" "name"="eedc4e94-592d-4857-aef2-4905ecdf75d9" "namespace"="default"
E0506 00:01:25.482172 1 manager.go:506] manager "msg"="Failed to issue certificate, retrying after applying exponential backoff" "error"="waiting for request: certificaterequest.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\" not found" "volume_id"="csi-d00aa30048a62abe239705e4427130b3f8b9bec2b54487c35153d9a5ca3db6c1"
Logs from cert-manager:
I0506 00:00:23.433363 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Approved" to 2022-05-06 00:00:23.433346151 +0000 UTC m=+1446937.641072660
I0506 00:00:23.456865 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.456854023 +0000 UTC m=+1446937.664580514
I0506 00:00:23.456913 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.456904982 +0000 UTC m=+1446937.664631475
I0506 00:00:23.457423 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.457415688 +0000 UTC m=+1446937.665142182
I0506 00:00:23.457598 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.457589161 +0000 UTC m=+1446937.665315652
I0506 00:00:23.457627 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.457613946 +0000 UTC m=+1446937.665340437
I0506 00:00:23.492341 1 controller.go:161] cert-manager/certificaterequests-issuer-ca "msg"="re-queuing item due to optimistic locking on resource" "key"="default/eedc4e94-592d-4857-aef2-4905ecdf75d9" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\": the object has been modified; please apply your changes to the latest version and try again"
I0506 00:00:23.499850 1 controller.go:161] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "key"="default/eedc4e94-592d-4857-aef2-4905ecdf75d9" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\": the object has been modified; please apply your changes to the latest version and try again"
I0506 00:00:23.500874 1 controller.go:161] cert-manager/certificaterequests-issuer-vault "msg"="re-queuing item due to optimistic locking on resource" "key"="default/eedc4e94-592d-4857-aef2-4905ecdf75d9" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\": the object has been modified; please apply your changes to the latest version and try again"
I0506 00:00:23.501078 1 controller.go:161] cert-manager/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to optimistic locking on resource" "key"="default/eedc4e94-592d-4857-aef2-4905ecdf75d9" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\": the object has been modified; please apply your changes to the latest version and try again"
I can't really tell what's happening here, tbh. It looks like the certificate -is- being issued, but not in time? So the CSI driver deletes the certificate request and tries again which again times out?
I'm using the Cloudflare DNS challenge solver for cert-manager, so it's possible that the propagation delay is to blame here. I've also tried it with the disable-auto-renew attribute.
ETA: after messing around with this for a bit, I eventually gave up and mounted the TLS secret as a volume on the container.
from csi-driver.
ETA: after messing around with this for a bit, I eventually gave up and mounted the TLS secret as a volume on the container.
@glmdev I have the same issue, what did you do to fix it?
from csi-driver.
For anyone running into the above, can they please share what the status is of the CertificateRequest which is causing them problems? e.g.:
kubectl describe cr eedc4e94-592d-4857-aef2-4905ecdf75d9
Also setting a higher log level on the csi-driver should help give an idea on what is happening:
--log-level=5
from csi-driver.
It sounds like everyone here is using Let's Encrypt with the CSI driver - is that correct?
I am not sure if the CSI driver is the best tool for this job if so, as the CSI driver will perform a whole new Order on every pod creation for a new certificate, which is likely going to eat up all your quota with their API.
Also to clarify, you don't need to use the CSI driver at all if you are creating a Certificate resource - you'll just want to mount the generated Secret into the Pod as a volume like you've resorted to above 😊
from csi-driver.
For now I fell back to manual certificate management, I'll check that when I get the time.
No, I have a local CA generated with openssl.
from csi-driver.
I am trying to use csi-driver
for pod certs (mTLS). I am also getting similar error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 46m default-scheduler Successfully assigned mtls-demo/nginx-csi-app to worker-1
Warning FailedMount 10m (x16 over 44m) kubelet Unable to attach or mount volumes: unmounted volumes=[tls], unattached volumes=[tls kube-api-access-l5gxv]: timed out waiting for the condition
Warning FailedMount 4m10s (x23 over 45m) kubelet MountVolume.SetUp failed for volume "tls" : rpc error: code = Unknown desc = timed out waiting for the condition
cert-manager log:
I0819 12:46:49.655107 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Approved" to 2022-08-19 12:46:49.655085489 +0000 UTC m=+20753.641048167
I0819 12:46:49.689897 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.689824879 +0000 UTC m=+20753.675787511
I0819 12:46:49.690161 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.690136449 +0000 UTC m=+20753.676099083
I0819 12:46:49.690568 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.690537081 +0000 UTC m=+20753.676499702
I0819 12:46:49.690728 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.690684788 +0000 UTC m=+20753.676647418
I0819 12:46:49.690923 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.690901191 +0000 UTC m=+20753.676863811
I0819 12:46:49.775161 1 controller.go:161] cert-manager/certificaterequests-issuer-venafi "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cc574045-998f-400a-87f6-c3641ada4bc3\": the object has been modified; please apply your changes to the latest version and try again" "key"="mtls-demo/cc574045-998f-400a-87f6-c3641ada4bc3"
I0819 12:46:49.783453 1 controller.go:161] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cc574045-998f-400a-87f6-c3641ada4bc3\": the object has been modified; please apply your changes to the latest version and try again" "key"="mtls-demo/cc574045-998f-400a-87f6-c3641ada4bc3"
I0819 12:46:49.802477 1 controller.go:161] cert-manager/certificaterequests-issuer-vault "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cc574045-998f-400a-87f6-c3641ada4bc3\": the object has been modified; please apply your changes to the latest version and try again" "key"="mtls-demo/cc574045-998f-400a-87f6-c3641ada4bc3"
I0819 12:46:49.822315 1 controller.go:161] cert-manager/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cc574045-998f-400a-87f6-c3641ada4bc3\": the object has been modified; please apply your changes to the latest version and try again" "key"="mtls-demo/cc574045-998f-400a-87f6-c3641ada4bc3"
$ kubectl describe cr cc574045-998f-400a-87f6-c3641ada4bc3 -n mtls-demo
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IssuerNotFound 85s cert-manager-certificaterequests-issuer-ca Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
Normal cert-manager.io 85s cert-manager-certificaterequests-approver Certificate request has been approved by cert-manager.io
Normal IssuerNotFound 85s cert-manager-certificaterequests-issuer-acme Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
Normal IssuerNotFound 85s cert-manager-certificaterequests-issuer-selfsigned Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
Normal IssuerNotFound 85s cert-manager-certificaterequests-issuer-vault Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
Normal IssuerNotFound 85s cert-manager-certificaterequests-issuer-venafi Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
From what I understood from the doc, one doesn't need to create any Certificate resource (in which we define issuerRef
kind as ClusterIssuer
) explicitly. I still tried creating certificate (skipped specifying dnsNames
) assuming that DNS name is already specified in the pod spec (csi.cert-manager.io/dns-names: ${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local
) but then my certificate failed with error which was actually expected:
admission webhook "webhook.cert-manager.io" denied the request: spec: Invalid value: "": at least one of commonName, dnsNames, uris ipAddresses, or emailAddresses must be set
Certificate:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: csi-example-com
namespace: mtls-demo
spec:
secretName: csi-example-com-tls
isCA: false
issuerRef:
name: ca-issuer
name: letsencrypt-staging
kind: ClusterIssuer
I will really appreciate any help.
from csi-driver.
Thanks, actually the name for the ClusterIssuer was incorrectly set. Now I am getting this error:
E0819 14:52:36.083386 1 sync.go:273] cert-manager/orders "msg"="failed to create Order resource due to bad request, marking Order as failed" "error"="400 urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for \"nginx-csi-app.mtls-demo.svc.cluster.local\": Domain name does not end with a valid public suffix (TLD)" "resource_kind"="Order" "resource_name"="9e3438d9-f5ff-4eb5-937a-a64f49f41d5a-1859061661" "resource_namespace"="mtls-demo" "resource_version"="v1"
Since .local
is not reachable, I am wondering how the csi-driver is expected to work for pods then? I did check the docs for csi-driver and read that it doesn't support self-signed certs but for letsencrypt, it doesn't say such thing. I did read @munnerz comment above that csi-driver with letsencrypt is not really how it's meant to be used due to quota limit but I at least wanted to see this working with .local
so I could then later use the same with real CA, venafi etc.
from csi-driver.
Related Issues (20)
- Receiving timeout error on Pod HOT 2
- Support all subject attributes
- JKS support HOT 1
- [deleted]
- Unable to get CSINode registered properly in k3s HOT 1
- Volume empty HOT 2
- SubPath support is broken or missing
- Update images to not utilize k8s.gcr.io HOT 2
- Push new tag for chart fixes HOT 1
- Release Helm Chart v0.5.1 / v0.6.0 HOT 4
- Add explicit namespace to all namespaced resources in Helm chart HOT 1
- E2E Test Cleanup HOT 1
- [Question]: how to set issuer to AWSPCAClusterIssuer HOT 6
- Feature Request: Add volumeAttributes to the generated CertificateRequest HOT 1
- Feature Request: Plase support setting the owner, group and permissions of TLS volume HOT 1
- rpc error: code = Unknown desc = mkdir /mnt: read-only file system HOT 3
- Missing cert-manager.io/revision-history-limit volume attributes for CSI-Driver
- Broken comma-separated splitting logic
- Certificate renewal doesn't change file 'modified date'
- Does cert-manager-csi-driver support AWS EKS with AWS Fargate nodes? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from csi-driver.