Pod Identity Association not recognised by secrets store CSI driver about secrets-store-csi-driver-provider-aws

[jbct]

Thank you for the report, we will look into this issue.

[steven-so]

I'm also getting:
Warning FailedMount 6s (x6 over 22s) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/user-service-depl-858474494c-5nr6w, err: rpc error: code = Unknown desc = us-west-2: An IAM role must be associated with service account default (namespace: default)

however I'm not using Pod Identity add-on.

I'm trying to spin up a private docker hub image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
    spec:
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets"
      containers:
        - name: my-service
          image: myprivate/dockerimage
          env:
            - name: JWT_KEY
              valueFrom:
                secretKeyRef:
                  name: jwt-secret
                  key: JWT_KEY
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
      imagePullSecrets:
        - name: "docker-hub"

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  secretObjects:
    - secretName: jwt-secret
      type: Opaque
      data:
        - objectName: jwt-secret
          key: JWT_KEY
    - secretName: docker-hub
      type: kubernetes.io/dockerconfigjson
      data:
        - objectName: "docker-configjson"
          key: ".dockerconfigjson"
  parameters:
    region: us-west-2
    objects: |
      - objectName: "jwt-secret"
        objectType: "secretsmanager"
      - objectName: "docker-configjson"
        objectType: "secretsmanager"

[gustavclausen]

This is must likely due to the provider using an older version of the AWS SDK (1.47.10) which doesn't support the container credential provider (supporting the EKS Pod Identity functionality). The minimum required SDK version is v1.47.11 (see https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html). Thus, this is also a thing to consider.

[ran2806]

I am also facing the same issue. Is there any solution on it?

[egorksv]

I am also facing the same issue. Is there any solution on it?

As a workaround I'm using good old OIDC-mapped IAM role, but would definitely want to move on to the new way.

[steven-so]

I am also facing the same issue. Is there any solution on it?

As a workaround I'm using good old OIDC-mapped IAM role, but would definitely want to move on to the new way.

What role does the service account need to "MountVolume.SetUp failed for volume 'secrets-store-inline'"?

[kevarr]

To offer a potential use-case: I'd like to use a cross-account IAM role to centralize permission management to a SecretsManager secret, use EKS Pod Identities to allow a Pod running in one account (Account A) to assume the centralized role in another account (Account B), and mount the secret into the Pod using secrets-store-csi-driver-provider-aws.

With EKS Pod Identities you can only associate roles that are in the same AWS account as the cluster. The documentation states that to achieve cross-account access you should use role chaining[1]. This is in contrast to IRSA which allows you to directly assume a role in another account [2].

If I were to use the provider to mount the secret into my Pod it would need to assume the role in Account B using the credentials of Account A before making the GetSecretValue API request.

[1] https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html
[2] https://aws.amazon.com/blogs/containers/cross-account-iam-roles-for-kubernetes-service-accounts/

[dprangnell]

I haven't tested this, but it's almost definitely resolved in the latest release 0.3.6 which deploys v1.49.19 of the SDK.

[dprangnell]

Well I spoke too soon, I tested it and it doesn't work with release 0.3.6:

  Warning  FailedMount  40s (x11 over 6m52s)  kubelet MountVolume.SetUp failed for volume "secret-from-secret-manager" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/demo-app, err: rpc error: code = Unknown desc = us-east-1: An IAM role must be associated with service account pod-identity (namespace: default)

helm list -n kube-system      
NAME                        	NAMESPACE  	REVISION	UPDATED                             	STATUS  	CHART                                      	APP VERSION
csi-secrets-store           	kube-system	1       	2024-02-17 10:52:16.835931 -0800 PST	deployed	secrets-store-csi-driver-1.4.1             	1.4.1      
secrets-provider-aws        	kube-system	1       	2024-02-17 10:52:34.524537 -0800 PST	deployed	secrets-store-csi-driver-provider-aws-0.3.6	

    spec:
      serviceAccountName: pod-identity
      volumes:
        - name: secret-from-secret-manager
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets"
      containers:
      - name: demo-app
        image: "example/demo-app:latest"
        volumeMounts:
        - name: secrets-from-secret-manager
          mountPath: "/mnt/secrets-store"
          readOnly: true
        imagePullPolicy: Always

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:{{.Values.region}}:{{.Values.accountid}}:secret:{{.Values.secretname}}"

[Fatma-J]

@dprangnell Same for me.
Have you succeeded in fixing the issue?

[giedriuskilcauskas]

That specific check for annotation is here: https://github.com/aws/secrets-store-csi-driver-provider-aws/blob/main/auth/auth.go#L110-L126
And I think there is no way to workaround it with parameters without code change.

[DanielMcAssey]

Hitting this issue too, unfortunately OIDC is a pain with its limitations on roles

[jbct]

Thank you for the request. The EKS POD Identities page calls out incompatibility with other CSI storage drivers and we are working to get documentation updated to include the Secrets Manager and Config Provider for Secret Store. We have this in our backlog and have marked it as a future enhancement.

[DanielMcAssey]

I have secrets manager manually installed, that page indicates that it should work, but I am hitting the same issue as the above

[yambottle]

System Info

• k8s Version: v1.28.6, created by kOps with EC2 instances(not EKS)
• cilium: quay.io/cilium/operator:v1.13.10
• pod-identity-webhook: amazon/amazon-eks-pod-identity-webhook:v0.4.0
• csi-secrets-store: 1.4.2
• secrets-store-csi-driver-provider-aws: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-58-g4ddce6a-2024.01.31.21.42 from doc

Setup Steps

• kOps created cluster already existis before this test, it has been created with Service Account Issuer Discovery and AWS IAM Roles for Service Accounts (IRSA)
• Installed csi-secrets-store and secrets-store-csi-driver-provider-aws following the doc
• Use kOps to provision IAM role and policy for service account:

spec:
  iam:
    serviceAccountExternalPermissions:
      - name: default-secret-manager
        namespace: default
        aws:
          inlinePolicy: |-
            [
              {
                "Effect": "Allow",
                "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
                "Resource": ["arn:*:secretsmanager:*:*:secret:*"]
              }
            ]

• Create SA default-secret-manager
• Create SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:                    # provider-specific parameters
    region: xxxx
    objects:  |
      - objectName: "xxxxx"
        objectType: "secretsmanager"

• Create Test Pod:

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  serviceAccount: default-secret-manager
  containers:
  - name: aws-cli
    image: amazon/aws-cli:latest
    command: ["/bin/bash", "-c", "tail -f /dev/null"]
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: test-secret
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: test-secret
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secrets"
  restartPolicy: Always

• Got pod stuck on ContainerCreating with error: MountVolume.SetUp failed for volume "test-secret" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/aws-cli, err: rpc error: code = Unknown desc = xxxx: An IAM role must be associated with service account default-secret-manager (namespace: default)
• Tried to attach SecretsManagerReadWrite policy, got the same error
• If I manually attach AWSS3ReadOnly policy to the kOps provisioned role, and comment out the volume mount

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  serviceAccount: default-secret-manager
  containers:
  - name: aws-cli
    image: amazon/aws-cli:latest
    command: ["/bin/bash", "-c", "tail -f /dev/null"]
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

I'm able to access the pre-created secret with the same service account

bash-4.2# aws secretsmanager get-secret-value --secret-id xxxxx
{
    "ARN": "arn:aws:secretsmanager:xxx:xxx:secret:xxx",
    "Name": "xxxx",
    "VersionId": "xxxxx",
    "SecretString": "xxxxxx,
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2024-03-19T21:00:25.681000+00:00"
}

Hope this info could help to resolve this issue sooner!

[giedriuskilcauskas]

• Create SA default-secret-manager

@yambottle you need to put annotation on SA with the role created in previous step.