blob-fuse with workload identity with version 1.23.3 not store file in azure storage account about blob-csi-driver

[cvvz]

@Xaviou1003 Currently AKS managed blob csi driver doesn't support this feature yet, so, if you want to try this feature, you have to disable the managed blob csi driver by running az aks update --disable-blob-driver first and then install the open-source blob csi driver manually by following the steps above. We will support this feature in managed blob csi driver from AKS v1.29.

[Xaviou1003]

After I have installed csi driver with chart helm, the csi driver works perfectly.

For your information when I installed csi driver the first time, I disabled the default aks blob driver with az aks update --disable-blob-driver.

I followed the installation guideline that provide here but csi driver seems not to be up to date.

And I see you just fix this problem with this pr.

@andyzhangx and @cvvz Thanks a lot for your help.

[andyzhangx]

/kind bug

[andyzhangx]

@Xaviou1003 have you specified serviceAccountName in your pod spec? if yes, could you paste all spec you have set?

[Xaviou1003]

@andyzhangx Thanks for your answer.

Yes I provided serviceAccountName in my pod dspec.

You can find my spec below (I just mask my client id):

---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  name: csi-blob
  namespace: cloudplatform-backend
  annotations: 
    azure.workload.identity/client-id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  labels:
    azure.workload.identity/use: "true"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: blob.csi.azure.com
  name: pv-blob
  namespace: cloudplatform-backend
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: blob-fuse
  mountOptions:
    - -o allow_other
    - --file-cache-timeout-in-seconds=120
  csi:
    driver: blob.csi.azure.com
    # make sure volumeid is unique for every storage blob container in the cluster
    # the # character is reserved for internal use, the / character is not allowed
    volumeHandle: rg#stgmakspocvolumetst#pvctest
    volumeAttributes:
      storageAccount: stgmakspocvolumetst
      containerName: pvctest  # required
      resourceGroup: rg-aks-tst # optional, specified when the storage account is not under AKS node resource group(which is prefixed with "MC_")
      clientID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # required
      # tenantID: $IDENTITY_TENANT  #optional, only specified when workload identity and AKS cluster are in different tenant
      # subscriptionid: $SUBSCRIPTION #optional, only specified when workload identity and AKS cluster are in different subscription


---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-blob
  namespace: cloudplatform-backend
  labels:
    app: nginx
    azure.workload.identity/use: "true"
spec:
  serviceName: statefulset-blob
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: csi-blob  #required, Pod does not use this service account has no permission to mount the volume
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: statefulset-blob
          image: artifactory-tst.tkub-tst3d5.az.groupemutuel.ch/docker/ops/alpine-gm:3.17
          lifecycle: 
            preStop: 
              exec: 
                command: 
                  - "/bin/sh"
                  - "-c"
                  - rm /mnt/blob/*
          command:
            - "/bin/sh"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/blob/outfile; sleep 1; done
          volumeMounts:
            - name: pv-blob
              mountPath: /mnt/blob
              readOnly: false
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: pv-blob
      spec:
        storageClassName: blob-fuse
        accessModes: ["ReadWriteMany"]
        resources:
          requests:
            storage: 10Gi

[Xaviou1003]

Hello @andyzhangx,

Have you had time to look at this problem?

[andyzhangx]

/assign @cvvz

[cvvz]

@Xaviou1003 Could you share the full logs of csi driver? The log message Skip NodeStageVolume for volume(rg#xxxxxx#xxxx) since clientID xxxxxxxx-xxxx-xxxx-xxxxxxxxx is provided but service account token is empty. is normal and we need more logs to investigate what exactly happened.

[Xaviou1003]

@cvvz yes, of course.

The full log of csi-blob-node:

utils.go:110] GRPC response: {}
nodeserver.go:156] NodePublishVolume: volume rg#stgmakspocvolumetst#pvctest mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount at /var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount successfully
mount_linux.go:218] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount /var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount)
mount_linux.go:218] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount /var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount)
nodeserver.go:140] NodePublishVolume: volume rg#stgmakspocvolumetst#pvctest mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount at /var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount with mountOptions: [bind]
utils.go:104] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount","target_path":"/var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120"]}},"access_mode":{"mode":5}},"volume_context":{"clientID":"xxxxxxxx-xxxx-xxxx-xxxxxxxxx","containerName":"pvctest","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"statefulset-blob-0","csi.storage.k8s.io/pod.namespace":"cloudplatform-backend","csi.storage.k8s.io/pod.uid":"4f0c9e90-26a9-4033-83ce-301a30fdea0e","csi.storage.k8s.io/serviceAccount.name":"csi-blob","resourceGroup":"rg-aks-tst","storageAccount":"stgmakspocvolumetst"},"volume_id":"rg#stgmakspocvolumetst#pvctest"}
utils.go:103] GRPC call: /csi.v1.Node/NodePublishVolume
utils.go:110] GRPC response: {}
nodeserver.go:256] Skip NodeStageVolume for volume(rg#stgmakspocvolumetst#pvctest) since clientID xxxxxxxx-xxxx-xxxx-xxxxxxxxx is provided but service account token is empty
utils.go:104] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120"]}},"access_mode":{"mode":5}},"volume_context":{"clientID":"xxxxxxxx-xxxx-xxxx-xxxxxxxxx","containerName":"pvctest","resourceGroup":"rg-aks-tst","storageAccount":"stgmakspocvolumetst"},"volume_id":"rg#stgmakspocvolumetst#pvctest"}
utils.go:103] GRPC call: /csi.v1.Node/NodeStageVolume

[cvvz]

Thanks, could you also get the kubelet logs?

[cvvz]

And please run kubectl get csidriver and paste the output

[Xaviou1003]

kubectl get csidriver
NAME                 ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES                  AGE
blob.csi.azure.com   false            true             false             <unset>         false               Persistent,Ephemeral   13d
csi.tigera.io        true             true             false             <unset>         false               Ephemeral              289d
disk.csi.azure.com   true             false            false             <unset>         false               Persistent             448d

kubelet log :

reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pv-blob\" (UniqueName: \"kubernetes.io/csi/blob.csi.azure.com^rg#stgmakspocvolumetst#pvctest\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
reconciler_common.go:228] "operationExecutor.MountVolume started for volume \"pv-blob\" (UniqueName: \"kubernetes.io/csi/blob.csi.azure.com^rg#stgmakspocvolumetst#pvctest\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
operation_generator.go:662] "MountVolume.MountDevice succeeded for volume \"pv-blob\" (UniqueName: \"kubernetes.io/csi/blob.csi.azure.com^rg#stgmakspocvolumetst#pvctest\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount\"" pod="cloudplatform-backend/statefulset-blob-0"
operation_generator.go:744] "MountVolume.SetUp succeeded for volume \"pv-blob\" (UniqueName: \"kubernetes.io/csi/blob.csi.azure.com^rg#stgmakspocvolumetst#pvctest\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
operation_generator.go:744] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2mq\" (UniqueName: \"kubernetes.io/projected/f5e37abd-3eae-4e9e-a03c-af11854f1306-kube-api-access-hk2mq\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
operation_generator.go:744] "MountVolume.SetUp succeeded for volume \"azure-identity-token\" (UniqueName: \"kubernetes.io/projected/f5e37abd-3eae-4e9e-a03c-af11854f1306-azure-identity-token\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cloudplatform-backend/statefulset-blob-0"
kubelet.go:2231] "SyncLoop (PLEG): event for pod" pod="cloudplatform-backend/statefulset-blob-0" event=&{ID:f5e37abd-3eae-4e9e-a03c-af11854f1306 Type:ContainerStarted Data:49bb4893c7da67ed2da1ce317818f74bc2bbfcfeaf957f941e55f13649912f69}
kubelet.go:2231] "SyncLoop (PLEG): event for pod" pod="cloudplatform-backend/statefulset-blob-0" event=&{ID:f5e37abd-3eae-4e9e-a03c-af11854f1306 Type:ContainerStarted Data:2244679131653363f75aa491b9e14296fa42ade4e8e053769d1906fb7305e5f2}

[cvvz]

You didn't install blob csi driver correctly. If the csi driver was installed correctly, then the TOKENREQUESTS field should be api://AzureADTokenExchange, it was unset currently.

kubectl get csidriver
NAME                 ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   **TOKENREQUESTS**   REQUIRESREPUBLISH   MODES                  AGE
blob.csi.azure.com   false            true             false             **<unset>**         false               Persistent,Ephemeral   13d

You can uninstall the existing blob csi driver first and use below steps to re-install blob csi driver and try again:

helm repo add blob-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/charts
helm repo update blob-csi-driver

helm install blob-csi-driver blob-csi-driver/blob-csi-driver \
--set node.enableBlobfuseProxy=true \
--set node.blobfuseProxy.installBlobfuse=false \
--set node.blobfuseProxy.installBlobfuse2=false \
--namespace kube-system \
--version v1.23.3

[andyzhangx]

@Xaviou1003 if you have installed open source driver, then you could use following command to update blob.csi.azure.com csidriver object, and then retry should work.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/csi-blob-driver.yaml