Failed mount after kubernetes worker node upgrade from v1.23.15 to v1.24.9 about beegfs-csi-driver

[mceronja]

Hi,

I upgraded beegfs-csi-driver to v1.4.0 and I removed /pv from plugins-mount-dir and it works now. I guess the upgrade of the version did the trick.

tree -L 4
.
├── beegfs.csi.netapp.com
│   └── 443016c18d78ef863d31e5904c346489c800bf6f3014713ce8a694a0fdce7bd6
│       ├── globalmount
│       │   ├── beegfs-client.conf
│       │   └── mount
│       └── vol_data.json
└── pv
    ├── pvc-0023367f
    │   ├── globalmount
    │   │   ├── beegfs-client.conf
    │   │   └── mount
    │   └── vol_data.json
...

I will close the issue.

Thanks!

[ejweber]

Thanks for opening this issue @mceronja. Apologies for the delayed response. It appears you have resolved the issue, but I thought I'd provide a bit of color anyway.

Kubernetes changed the staging paths for persistent volumes in kubernetes/kubernetes#107065. Because they are no longer staged under .../csi/pv (and instead under the more general .../csi), and our old deployment manifests only give the driver purview over .../csi/pv, our old manifests are not compatible with Kubernetes >=1.24. That being said, our updated manifests (since >=1.3.0) use the more general .../csi and should be backwards compatible (as .../csi/pv is a subdirectory of .../csi).

It's hard to understand the exact cause of your issue without diving in pretty deep, but it seems likely to be upgrade flow related. It would make a lot more sense to me if you were coming from v1.2.2 (which had incompatible base manifests), but that doesn't appear to be the case. To anyone else arriving here for some similar reason:

1. Be sure you are using the base manifests and overlays as described in https://github.com/NetApp/beegfs-csi-driver/tree/master/deploy/k8s#basics so you will always pick up the latest base manifests when upgrading the driver.
2. Do not attempt to use driver version <1.3.0 with Kubernetes version >=1.24.0, as there is a hard incompatibility related to the discussion here.