Cannot mount secrets to use as ImagePullSecrets about secrets-store-csi-driver-provider-aws

[JRemitz]

Okay... NM, I think it is working now. @miki79 - not sure if you've since given up on this but the docs I linked above might have been outdated last year? I noticed that earlier this year I upgraded to v0.3.0 and there are newer versions.

1. I upgraded my Secrets Store CSI Driver to v1.0.1 (note, they just released v1.1.0 but I'm not on that yet). I don't see a release in the chart yet.
2. I upgraded my secret to apiVersion: secrets-store.csi.x-k8s.io/v1
3. After updating my pod I'm able to connect. I see the docker secret created and no events logged for an invalid ImagePullSecret.

More info for anyone else trying from scratch:

1. AWS SSM - SecureString - contains JSON format of the docker-registry secret
2. I have a ServiceAccount annotated with the correct role to the SSM secure string AND KMS key to decrypt the value (if needed)
3. My secret looks like the example in the issue, copied here for reference:

   ---
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: docker-hub
   spec:
     provider: aws
     secretObjects:
       - secretName: docker-hub
         type: kubernetes.io/dockerconfigjson
         data:
           - objectName: "docker-configjson"
             key: ".dockerconfigjson"
     parameters:
       region: us-east-1
       objects: |
         - objectName: "/secret/docker_config"
           objectAlias: docker-configjson
           objectType: "ssmparameter"

4. Volume and Mounts like so:

   volumes:
     - name: secrets-store-inline
       csi:
         driver: secrets-store.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: "docker-hub"
   
   volumeMounts:
     - name: secrets-store-inline
       mountPath: "/mnt/secrets-store"
       readOnly: true

5. ServiceAccount attached to pod
6. ImagePullSecrets added for secret reference:

   imagePullSecrets:
     - name: "docker-hub"

[Luis-DevOps]

All along I was missing the --set syncSecret.enabled=true for the driver installation. It just worked after that installation. I'm good now.

helm install -n kube-system csi-secrets-store
--set syncSecret.enabled=true
secrets-store-csi-driver/secrets-store-csi-driver

[Tommyf]

Thanks so much for that. You homed right in on the bit I was missing - that the K8s secret is created with the volume creation as opposed to the volume mount.

I was doing a bit of a mix of implementation and research. Yes, I am trying to implement this, but I can't implement things without first trying my best to understand how they work.

No, my implementation was not working. I was trying to understand the process and decided that it did not make sense how it worked, so I was homing in on that problem rather than finding my - ahem - typo :). It works now.

That all said, there is still a bit I don't get. I tried experimenting to include the volumes in my pod spec, but not include the volume mounts. If what you say, (and my current understanding) is correct, then I would expect the volume to still get created in this scenario and then the password could still be synced. But that did not work. No volume mount - no K8s secret.

Maybe the tool is looking to see if a volumeMount exists in the spec before syncing the secret, as opposed to waiting for the mount to become realized. That would make sense, but much as I would like to get a deeper understanding of this, I don't have time to dig through the code right now.

The above test was primarily an exercise in me trying to build my understanding of the process, but in addition to that, it would be nice to not be exposing my private repo credentials inside the container. That seems unnecessarily exposed.

I guess the solution to this, without exposing the registry secret to the running containers, is to have some utility pod/sidecar that mounts the secret that contains said credentials.

Still seems a little hacky :/

[lasred]

Thank you for opening this issue - we are looking into it.

[yossicohn-hs]

Have a similar issue.
The Secrets are mounted to /mnt/secrets-storeas expected, but the secret (no matter of the type) that should have been created is not created.
Is there any way I can debug this issue ?

[asonnleitner]

same

[brucedvgw]

Facing the same issue. I thought I had incorrect configuration #38

[JRemitz]

Any updates on this @lasred? According to the secrets store CSI driver docs, the type kubernetes.io/dockerconfigjson is supported.

[miki79]

@JRemitz I trust you, I haven't investigated further as we found a different way so we are not using csi for docker secrets anymore.

[Luis-DevOps]

@JRemitz for the life of me I can't get this to actually pull the image. I get as far as mounting the secret with the CSI driver, but there must be something off at the parameter store that I'm not doing right I think.
Can you provide an example of this here: 1. AWS SSM - SecureString - contains JSON format of the docker-registry secret

[JRemitz]

Sure thing, @Luis-DevOps . This is what that SSM secret located at /secret/docker_config would look like:

{
  "auths": {
    "index.docker.io": {
      "username": "docker-user",
      "password": "docker-password",
      "email": "[email protected]",
      "auth": "<base64 of 'username:password' above>"
    }
  }
}

So the value of auth is the result of echo "username:password" | base64

[Luis-DevOps]

@JRemitz That's interesting. That's actually exactly what I've been doing. So I'm still in the same place. It's very puzzling right now.

[Tommyf]

This issue confuses me. I don't see how it's possible that this can work.

It's my understanding that in order to sync the secret to a Kubernetes secret, the container needs to come up so that the volumes can be mounted.

But in order for that to happen, it first needs to pull the container. For that to happen, it needs the image pull secret. And for that to happen it needs to sync.

And now we have a circular dependency, unless I am missing something fundamental about how this works.

I guess you could circumvent that problem by having some other pod( or sidecar) running that uses the same service account and an image from a repository that needs no auth. That pod then mounts the AWS Secrets Manager secret and the K8s secrets can then be synced.

That is a lot of extra mucking about and relies on the organisation allowing anonymous image pulling.

Am I missing something here?

[JRemitz]

Are you trying to get it to work and can't or just researching? I have to assume it is the order of operations on the details of what is happening between the volume creation and the mounting. So likely what you're missing is the volume creation, versus the volume mounting. The secret gets created with the volume not the mount. If it were an environment variable within the container then it would first need to be mounted. Presumably, but I haven't dug through the code to explain with references.