auto-unseal gcpckms doesn't work about vault-helm

[jasonodonnell]

Hi @cofonseca, good question!

The project-dev.json file will need to be mounted as a secret to the Vault container.

# Create the secret
kubectl create secret generic project-dev --from-file=project-dev.json

Then you would add this to your config:

server:
  extraVolumes:
    - type: secret
      name: project-dev

Vault Helm will mount this to /vault/userconfig/project-dev/project-dev.json.

Hope that helps!

[devopsbarista]

Hi @SamolazovRoman
In case you're still trying to resolve this, had the same issue and was able to mitigate it by removing the GOOGLE_REGION and GOOGLE_PROJECT envs. For some reason they don't play nice with Region and Project inside the 'gcpckms' seal stanza. Also, in case you're planning to use GCS as a HA storage backend, keep in mind that GOOGLE_CREDENTIALS won't work and you have to use GOOGLE_APPLICATION_CREDENTIALS, which would work for both 'gcpckms' and 'gcs'.

[cofonseca]

I know this is closed, but I'm running into the same issue. In your config, you have your credential json file at this path: /vault/userconfig/project-dev/project-dev.json

I'm confused as to where this file is actually located. Is this a path inside of Vault, a path on my machine running Helm, a path in a storage bucket? Where am I supposed to put my credential file and how does it get into Vault?

[SamolazovRoman]

kubectl create secret generic project-dev --from-file=./project-dev.json

➜ infrastructure git:(DEV) gcloud kms keyrings list --location global | grep vault-init-test projects/project-dev/locations/global/keyRings/vault-init-test

➜ infrastructure git:(DEV) gcloud kms keys list --keyring=vault-init-test --location=global NAME PURPOSE ALGORITHM PROTECTION_LEVEL LABELS PRIMARY_ID PRIMARY_STATE projects/project-dev/locations/global/keyRings/vault-init-test/cryptoKeys/vault-unseal-key-test ENCRYPT_DECRYPT GOOGLE_SYMMETRIC_ENCRYPTION SOFTWARE 1 ENABLED

➜ infrastructure git:(DEV) gcloud iam service-accounts list | grep vault-gcpkms Vault KMS for auto-unseal [email protected] False

[SamolazovRoman]

I run the manifests(https://github.com/hashicorp/vault-helm/tree/master/test/terraform) and get the same errors:

Error parsing Seal configuration: failed to encrypt with GCP CKMS - ensure the key exists and the service account has at least roles/cloudkms.cryptoKeyEncrypterDecrypter permission: rpc error: code = InvalidArgument desc = Resource name [projects/project-vault,/locations/global,/keyRings/vault-helm-unseal-kr/cryptoKeys/vault-helm-unseal-key] does not match any known resource name pattern.

If I run without the block:

     seal "gcpckms" {
         project     = "project-vault"
         region      = "global"
         key_ring    = "vault-helm-unseal-kr"
         crypto_key  = "vault-helm-unseal-key"
      }

all working, but unseal from: vault operator unseal into vault container.

FYI:

➜  infrastructure git:(DEV) kubectl exec -it vault-0 sh
/ # ls -la /vault/userconfig/project-vault/
total 4
drwxrwxrwt    3 root     root           100 Aug 29 13:43 .
drwxr-xr-x    3 root     root          4096 Aug 29 13:43 ..
drwxr-xr-x    2 root     root            60 Aug 29 13:43 ..2019_08_29_13_43_54.509151465
lrwxrwxrwx    1 root     root            31 Aug 29 13:43 ..data -> ..2019_08_29_13_43_54.509151465
lrwxrwxrwx    1 root     root            24 Aug 29 13:43 project-vault.json -> ..data/project-vault.json
/ # ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh -ec sed -E "s/HOST_IP/${HOST_IP?}/g" /vault/config/extraconfig-from-values.hcl > /tmp/sto
    9 root      0:00 {docker-entrypoi} /usr/bin/dumb-init /bin/sh /usr/local/bin/docker-entrypoint.sh vault server -co
   10 vault     0:00 vault server -config=/tmp/storageconfig.hcl
   86 root      0:00 sh
  186 root      0:00 ps aux
/ # cat /tmp/storageconfig.hcl
ui = true

listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}

storage "consul" {
  path = "vault"
  address = "10.128.0.5:8500"
}

# seal "gcpckms" {
#    credentials = "/vault/userconfig/project-vault/project-vault.json"
#    project     = "project-vault"
#    region      = "global"
#    key_ring    = "vault-helm-unseal-kr"
#    crypto_key  = "vault-helm-unseal-key"
# }/ #

[jasonodonnell]

@SamolazovRoman Are you positive project is the correct name in your seal config?

You'll want to use PROJECT_ID from:

$ gcloud projects list

[SamolazovRoman]

hi @jasonodonnell,
I've a lot of projects and only one have differents with PROJECT_ID <=> NAME.
Basically PROJECT_ID = NAME.

[SamolazovRoman]

I installed and init gcloud into vault container, after run:

kubectl exec -it vault-0 -- bash -c "/root/google-cloud-sdk/bin/gcloud kms keys list --keyring=vault-helm-unseal-kr03 --location=global"
NAME                                                                                                              PURPOSE          ALGORITHM                    PROTECTION_LEVEL  LABELS  PRIMARY_ID  PRIMARY_STATE
projects/project-vault/locations/global/keyRings/vault-helm-unseal-kr03/cryptoKeys/vault-helm-unseal-key03  ENCRYPT_DECRYPT  GOOGLE_SYMMETRIC_ENCRYPTION  SOFTWARE                  1           ENABLED

FYI into vault container:

~/google-cloud-sdk/bin # ./gcloud init
Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:
  gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

Choose the account you would like to use to perform operations for
this configuration:
 [1] [email protected]
 [2] Log in with a new account
Please enter your numeric choice:  1

You are logged in as: [[email protected]].

Pick cloud project to use:
 [1] project-vault
 [2] Create a new project
Please enter numeric choice or text value (must exactly match list
item):  1

Your current project has been set to: [project-vault].

[SamolazovRoman]

@jasonodonnell ^^^

[SamolazovRoman]

@jasonodonnell history from vault container:

~/google-cloud-sdk/bin # history
   0 gcloud
   1 apk add --update \
   2  python \
   3  curl \
   4  which \
   5  bash
   6 curl -sSL https://sdk.cloud.google.com | bash
   7 gcloud init
   8 cd /root/google-cloud-sdk/bin/
   9 ls -la
  10 ./gcloud
  11 ./gcloud init
  12 ./gcloud kms keys list --keyring=vault-helm-unseal-kr03 --location=global
  13 pwd
  14 history

[SamolazovRoman]

Hi @znatokov and @jasonodonnell
I found the solution, when auto-unseal working, see the config file:

global:
  enabled: true

server:
  extraVolumes:
    - type: secret
      name: project-vault
      load: false
    - type: secret
      name: vault-tls
      load: false

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: {{ template "vault.name" . }}
              release: "{{ .Release.Name }}"
              component: server
          topologyKey: kubernetes.io/hostname

  service:
    enabled: true

  ha:
    enabled: true
    replicas: 3

    config: |
      ui = true
    
      listener "tcp" {
        tls_disable = false

        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-tls/vault.key"

        tls_disable_client_certs = true
      }

      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }

      seal "gcpckms" {
        credentials = "/vault/userconfig/project-vault/project-vault.json"
        project     = "project-vault"
        region      = "global"
        key_ring    = "vault-helm-unseal-kr"
        crypto_key  = "vault-helm-unseal-key"
      }

ui:
  enabled: true
  serviceType: LoadBalancer

[jasonodonnell]

@SamolazovRoman and @znatokov Thanks for the solutions! I will update the documentation to include credentials in the seal config.

[cofonseca]

@jasonodonnell, thank you so much, that was very helpful! I gave that a try and it worked perfectly. Greatly appreciate the quick response.

Take care!

[stevegore]

@cofonseca could that be added to the documentation somewhere please? I'm attempting the helm chart and this is the only place I could find this explained.

From a Kubernetes perspective it makes complete sense, but a pointer might help save people time.