rpc error: code = Unknown desc = mkdir /mnt: read-only file system about csi-driver

[ad-zsolt-imre]

I refactored NodePublishVolume method of csi-lib to be able to unit test specifically the volume mounting which helped to reproduce the issue in other envs. Indeed, the issue happens when the targetPath for the volume is a non-existing directory on a read-only file system.

Given that:

1. The CSI driver is running inside a container with a read-only filesystem (see yaml snippet below)
2. The /mnt directory does not exist inside the cert-manager-csi-driver container
3. For some unknown reason the request has targetPath set to /mnt for some (yet) unknown reason

The MkdirAll instruction will return with a permission error.

        - name: cert-manager-csi-driver
          securityContext:
            readOnlyRootFilesystem: true

Removing the read-only flag from the rootfs of the cert-manager-csi-driver container is obviously not a good idea. Thus, the only thing there is moving forward is figuring out where /mnt coming from and why...

[ad-zsolt-imre]

I think I found where /mnt is coming from. MkdirAll creates directories recursively, but in case of an error, e.g. permission issue, it will only report the first directory it failed. E.g. if targetPath is /mnt/data/kubelet/... and /mnt does not exist and it's a read-only FS, it will throw a permission error with message mkdir /mnt: read-only file system.

I logged onto the node and cd'd into the csi driver's root FS at /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/1f27f2ae202e7c4eb9037b48a2a4701df6b8ab7f03c9d97c2b9a0b4baf5d9cca/rootfs and created the /mnt directory. Then, started a pod and saw this time it was complaining about /mnt/data. Now that directory was indeed familiar to me.

ls -ld /var/lib/kubelet/pods
lrwxrwxrwx 1 root root 22 Apr 21 20:29 /var/lib/kubelet/pods -> /mnt/data/kubelet/pods

$ helm template jetstack/cert-manager-csi-driver
     ...
      volumes:
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/cert-manager-csi-driver
            type: DirectoryOrCreate
        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry
            type: Directory

All the volume dirs under /var/lib/kubelet/ live under /mnt/data in my environment has an external hard drive mounted dedicated to k3s pods.

/mnt/data on the node is not read-only. Setting readOnlyRootFilesystem to false (for testing purposes only) on the cert-manager-csi-driver container resolved the mounting problem. (However, the mounted filesystem is empty, it does not contain anything)

[ad-zsolt-imre]

The solution was to not to symlink directories but mount the external drive to /var/lib/kubelet directly.