auto-unseal returns denied for resource even with proper service account permissions about vault-helm

[jasonodonnell]

@mprinvale,

I just enabled workload identity on GKE, this definitely changed some configurations. Here's how I got it to work:

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT-NAME.svc.id.goog[KUBE-NAMESPACE/KUBE-VAULT-SERVICE-ACCOUNT-NAME]" \
  GOOGLE_SERVICE_ACCOUNT_NAME@[email protected]

Next, I made the following configurations:

global:
  enabled: true
  image: "vault:1.2.3"

server:
  extraEnvironmentVars:
    GOOGLE_REGION: us-east1
    GOOGLE_PROJECT: vault-helm-dev

  serviceaccount:
    annotations:
      iam.gke.io/gcp-service-account: GOOGLE_SERVICE_ACCOUNT_NAME@PROJECT_NAME.iam.gserviceaccount.com

  ha:
    enabled: true
    replicas: 3

    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "gcpckms" {
        project     = "REGION"
        region      = "PROJECT-NAME"
        key_ring    = "KEY-RING-NAME"
        crypto_key  = "KEY-NAME"
      }

      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }

There's no need to mount credentials when using workload identity. Give it a shot!

[jasonodonnell]

Hi @mprinvale,

I see the environment variables in your example have been commented out. You tried using all of these environment variables at once?

I'm not able to reproduce this on my end: Vault is auto unsealing using Google KMS from a GKE cluster using a service account with Cloud KMS CryptoKey Encrypter/Decrypter permissions. One thought was, does the key already exist within the keyring? If the key doesn't exist, it will require permissions to create it (KMS admin).

My deployment looks like this:

global:
  enabled: true
  image: "vault:1.2.3"

server:
  extraEnvironmentVars:
    GOOGLE_REGION: us-east1
    GOOGLE_PROJECT: vault-helm-dev
    GOOGLE_CREDENTIALS: /vault/userconfig/kms-creds/credentials.json
    GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/kms-creds/credentials.json

  extraVolumes:
    - type: "secret"
      name: "kms-creds"

  ha:
    enabled: true
    replicas: 3

    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "gcpckms" {
        project     = "us-east1"
        region      = "vault-helm-dev"
        key_ring    = "vault-helm-test"
        crypto_key  = "vault-helm-us-east"
      }

      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }

[mprinvale]

Hi Jason,

I have tried using all of them at once.

The keyring, key, and service account all exist currently and not trying to be created upon installation of vault. Our values.yaml for vault looks very similar to the one you posted with the addition of all the other default items that came with the 0.1.2 release.

Can you describe GKE and node pool setup? Perhaps something is miss configured. We have workload identity enabled on the GKE cluster and our node pool is configured to use the service account associated with the keyring/key which has owner permissions currently (while we debug this issue). All other GKE/NodePool options are default.

Thanks

[mprinvale]

Thanks I will first try with workload identity disabled. We enabled it per Google Support suggestions.

[mprinvale]

Thanks for the help @jasonodonnell. No change after disabling workload identity on the cluster. It is apparent something is missing but no idea what it could be.

[jasonodonnell]

@mprinvale,

Did you try with workload identity enabled and the solution I posted above? If you aren't using workload identity, you'll need to mount credentials into the Vault pod as originally suggested.

[mprinvale]

With workload identity disabled I was using my original values.yaml with the mounts and what not. I will try again w/ workload identity enabled and your suggested values.

[mprinvale]

@jasonodonnell - couple of questions:

• KUBE-NAMESPACE: would I use default, kube-public, kube-system, or vault0ns
• KUBE-VAULT-SERVICE-ACCOUNT-NAME: Is this the default GKE service account or the service account we provisioned and tied to KMS key?
• KMS_NAME: Not sure if this is supposed to be the key name? keyring name, etc

Thanks

[mprinvale]

@jasonodonnell - I enabled Workload Identity and went through steps to get it going as well as updated my values.yaml. I am at least getting a new error:

dorg-mbp:vault-helm-0.1.2 Matthew$ kubectl logs vault-0
Error parsing Seal configuration: failed to encrypt with GCP CKMS - ensure the key exists and the service account has at least roles/cloudkms.cryptoKeyEncrypterDecrypter permission: rpc error: code = Unauthenticated desc = Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

[jasonodonnell]

KUBE-NAMESPACE: would I use default, kube-public, kube-system, or vault0ns

This namespace where Vault is deployed.

KUBE-VAULT-SERVICE-ACCOUNT-NAME: Is this the default GKE service account or the service account we provisioned and tied to KMS key?

The service account attached to Vault (if you didn't change the default, it will be vault).

KMS_NAME: Not sure if this is supposed to be the key name? keyring name, etc

Keyring name, my apologies for the confusion!

[jasonodonnell]

Actually, KMS_NAME should be the name of your google service account. I followed the instructions here: https://cloud.google.com/blog/products/containers-kubernetes/introducing-workload-identity-better-authentication-for-your-gke-applications

[mprinvale]

I was able to use an outside support channel to help diagnose the issue. We think the issue came up because of a misconfigured key region. The keyring was in us-east1, thus set the stanza region to us-east1. After setting the region to global it seemed to have worked though we are still testing. I think a lot of future support issues could be avoided with a detailed step by step walkthrough of the setup.

[PayalSasmal10]

Hi Anyone can help me with these confusion that if my KMS keyrings should also be attached to the same GOOGLE_SERVICE_ACCOUNT_NAME ?