secretObjects not working as expected about secrets-store-csi-driver-provider-aws

[lenny]

I was having the same issues as @oyamin but I got it to work. There were multiple issues. First, commenting out the env parts in my deployment (to avoid downstream errors), I noticed the files were not getting mounted into the filesystem as per file named in jmesPath.objectAlias as they were in another cluster where secrets were working properly. I was able to resolve this by uninstalling/reinstalling the csi-secrets-store to their latest versions. Make sure all the csi related pods in the kube-system namespace roll over.

e.g.

helm -n kube-system uninstall csi-secrets-store
kubectl -n kube-system delete daemonsets/csi-secrets-store-provider-aws

Then:

$ helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts
$ helm -n kube-system upgrade -i csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true
$ curl -s https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml | kubectl apply -f -

After that, when I rolled over my application pods, they did have the mounted but the env variables were still wrong. This was also visible with kubectl describe secrets/<myappsecrets>. Explicitly deleting the k8 secret named in secretObjects.secretName and allowing it to be recreated did the trick.

[cryptk]

@fert-f you are almost there. Try something like this

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: csi-poc-aws-secrets
  namespace: kube-public
spec:
  secretObjects:                                
  - secretName: csi-poc
    type: Opaque
    labels:                                   
      provider: "csi-driver"
    data: 
    - objectName: csi-poc-username
      key: username
    - objectName: csi-poc-password
      key: password
  provider: aws
  parameters:
    objects: |
        - objectName: "/secrets/xxx/csi-poc"
          objectType: "secretsmanager"
          objectVersionLabel: "AWSCURRENT"
          objectAlias: csi-poc
          jmesPath:
          - path: "username"
            objectAlias: "csi-poc-username"
          - path: "password"
            objectAlias: "csi-poc-password"

Notice that rather than defining the secret twice, the secret is only defined a single time, just with two data items in it.

[oyamin]

I am almost there, but now I have separate file with correct credentials, but when I call my code, the env variables still have the full Json object:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secret-application
spec:
  provider: aws
  secretObjects:
    - secretName: csi-secrets
      type: Opaque
      labels:                                   
        provider: "csi-driver"
      data:
        - objectName: postgresql_database
          key: main_database
        - objectName: postgresql_host
          key: main_host
        - objectName: postgresql_user
          key: main_username
        - objectName: postgresql_password
          key: main_password
  parameters:
    objects: |
      - objectName: "rds-secrets"
        objectType: "secretsmanager"
        objectAlias: csi-secrets
        jmesPath: 
          - path: "main_database"
            objectAlias: "postgresql_database"
          - path: "main_host"
            objectAlias: "postgresql_host"
          - path: "main_username"
            objectAlias: "postgresql_user"
          - path: "main_password"
            objectAlias: "postgresql_password"

Which generates the files with correct values from each key:

# ls -ltr /mnt/secrets-store/
total 20
-rw-r--r-- 1 root root   8 Aug 31 21:16 postgresql_user  # has the value of postgresql_user key
-rw-r--r-- 1 root root  20 Aug 31 21:16 postgresql_password  # has the value of postgresql_password key
-rw-r--r-- 1 root root  48 Aug 31 21:16 postgresql_host  # has the value of postgresql_host key
-rw-r--r-- 1 root root   7 Aug 31 21:16 postgresql_database  # has the value of postgresql_database key
-rw-r--r-- 1 root root 156 Aug 31 21:16 csi-secrets  # contains full JSON object with all the key/values

When I call them in my Deploy, i still get each env variable that contains the full json:

postgresql_database={"main_database":"value","main_host":"value","main_password":"value","main_username":"value"}
postgresql_user={"main_database":"value","main_host":"value","main_password":"value","main_username":"value"}

deployment code:

      volumes:
        - name: api-secret
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secret-application"
      containers:
        - name: api
          image: "{{ .Values.images.api.image.repository }}"
          imagePullPolicy: Always
          ports:
            - containerPort: 80
          volumeMounts:
            - name: api-secret
              mountPath: "/mnt/secrets-store"
              readOnly: true
          env:
            - name: postgresql_database
              valueFrom:
                secretKeyRef:
                  name: csi-secrets
                  key: main_database
            - name: postgresql_host
              valueFrom:
                secretKeyRef:
                  name: csi-secrets
                  key: main_host
            - name: postgresql_user
              valueFrom:
                secretKeyRef:
                  name: csi-secrets
                  key: main_username
            - name: postgresql_password
              valueFrom:
                secretKeyRef:
                  name: csi-secrets
                  key: main_password

Not sure what else do try... or how to call those values from each file in the env

[atomicloopzilla]

Faced with similar issue recently - and spend couple of hours to find out..
I found that "jmesPath" should not contain any special characters like "-".
For example "main-database" wont work. But "main_database" will work.

[simonmarty]

Tracking in #66

[ArchiFleKs]

It is weird, this is working for me for example:

spec:
  parameters:
    objects: |
      - objectName: "/node/secret"
        objectType: "secretsmanager"
        objectAlias: "node-secret"
        jmesPath:
          - Path: "operator_key"
            ObjectAlias: "operator_key"
  provider: aws
  secretObjects:
  - data:
    - key: operator_key
      objectName: operator_key
    secretName: node-secret
    type: Opaque

Inside the secret key operator_key I get the content of the JSON path in operator_key

[fert-f]

Noted that this approach works when there is single secret key. When there are two only first one is rendered:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: csi-poc-aws-secrets
  namespace: kube-public
spec:
  secretObjects:                                
  - secretName: csi-poc
    type: Opaque
    labels:                                   
      provider: "csi-driver"
    data: 
    - objectName: csi-poc-username
      key: username
  - secretName: csi-poc
    type: Opaque
    labels:                                   
      provider: "csi-driver"
    data: 
    - objectName: csi-poc-password                   # name of the mounted content to sync. this could be the object name or object alias 
      key: password
  provider: aws
  parameters:
    objects: |
        - objectName: "/secrets/xxx/csi-poc"
          objectType: "secretsmanager"
          objectVersionLabel: "AWSCURRENT"
          objectAlias: csi-poc
          jmesPath:
          - path: "username"
            objectAlias: "csi-poc-username"
          - path: "password"
            objectAlias: "csi-poc-password"

Results in secret with key username only.

AWS SM is like:

{
  "username": "admin",
  "password": "abc123",
}

[fert-f]

@oyamin, I think you env should reference secret created by SecretProviderClass. Not sure where ngs-rds-secrets came from:

          env:
            - name: postgresql_database
              valueFrom:
                secretKeyRef:
                  name: csi-secrets
                  key: main_database
            - name: postgresql_host
              valueFrom:
                secretKeyRef:
                  name: csi-secrets
                  key: main_host
            - name: postgresql_user
              valueFrom:
                secretKeyRef:
                  name: csi-secrets
                  key: main_username
            - name: postgresql_password
              valueFrom:
                secretKeyRef:
                  name: csi-secrets
                  key: main_password

[oyamin]

@fert-f That was the old value from the original code. But all names where the same. It would not even deploy if the name in secretKeyRef didn't exist.

I ended up just doing a single env variable, and then rewrote the code to parse the json from that env var and get each key, but still would like to see if there is a solution for this.

[sharmavijay86]

Hi ! same problem facing here.

bellow config is working but all username and password in same inline value

working.

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider: aws
  secretObjects:
  - secretName: api-token
    labels:                                   
        provider: "csi-driver"
    data:
    - key: username
      objectName: api-token
    - key: password
      objectName: api-token
    type: Opaque
  parameters:
    objects: |
      - objectName: prod/service/token
        objectType: secretsmanager
        objectVersionLabel: "AWSCURRENT"
        objectAlias: api-token
        jmesPath:
        - path: username
          objectAlias: asmusername
        - path: password
          objectAlias: asmpassword

What i am getting is this, perhaps expected as objectName is top level object. But it is not working with jmesPath, where i am defining objectAlias, and using the same objectAlias to call in secret data.

USERNAME={"MY_API_TOKEN":"xxxxxxtest","username":"testuser","password":"testpass"}
PASSWORD={"MY_API_TOKEN":"xxxxxxtest","username":"testuser","password":"testpass"}

When i am using bellow config it does not creates secret.

notworking

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider: aws
  secretObjects:
  - secretName: api-token
    labels:                                   
        provider: "csi-driver"
    data:
    - key: username
      objectName: asmusername
    - key: password
      objectName: asmpassword
    type: Opaque
  parameters:
    objects: |
      - objectName: prod/service/token
        objectType: secretsmanager
        objectVersionLabel: "AWSCURRENT"
        objectAlias: api-token
        jmesPath:
        - path: username
          objectAlias: asmusername
        - path: password
          objectAlias: asmpassword

Any luck in same scenario with anyone ?

[Karabiy]

@sharmavijay86 I've ended up creating a bash scripts in order to be put alogside app execution on the docker entrypoint - it feels okay and gives you an option to have " secret key ref".
I do have an issue maybe related to yours, when I try to create an alias and then mount it I receive following issue

Warning  FailedMount  2s (x3 over 4s)  kubelet, ip-10-0-2-100.us-east-1.compute.internal  MountVolume.SetUp failed for volume "staging-app-alias" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod namespace/app-66b577db9d-wkw8r, err: rpc error: code = Unknown desc = Name already in use for objectAlias: staging-app-alias

From my side on currently I may recommend you to do the same, as changing app logic would be a very bad practice.
Sharing my bash scripts within the entrypoint_envs.sh

#!/bin/bash
ALL_ENVS=$(cat /mnt/secrets-store/*)
echo $ALL_ENVS
INPUT_DICT=$(echo $ALL_ENVS | tr -d "{}\"\n")
INPUT_DICT_CLEARED=$(sed "s/,/ /g" <<< $INPUT_DICT)
ROWS=$(echo $INPUT_DICT_CLEARED | sort )
echo $ROWS
IFS=" "
for ROW in ${ROWS}
    do
      RESULTED_ENV=$(sed "s/:/=/g" <<< $ROW)
      export $RESULTED_ENV
   done;

So entrypoint.sh would look like:

. entrypoint_envs.sh
app start

This one is case for

volumeMounts:
    - name: app-conf
      mountPath: "/mnt/secrets-store"
      readOnly: true
volumes:
    - name: "app-conf"
      csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
              secretProviderClass: "app-conf"

Can somebody provide full solution with nginx image for example, in case if he has a success with env variables split on the side of this one provider? @cryptk
It would be really easier to understand how it works if somebody has a success of it and how to debug local case.

[Karabiy]

Faced with similar issue recently - and spend couple of hours to find out.. I found that "jmesPath" should not contain any special characters like "-". For example "main-database" wont work. But "main_database" will work.

in my case that was /, though, - worked
somebody definitely should doc up this one moment