prometheus is not enabled% about vault-helm

[cablespaghetti]

The tricky bit which took me a while to work out is that the unauthenticated_metrics_access needs to be within your listener config e.g.

listener "tcp" {
  telemetry {
    unauthenticated_metrics_access = "true"
  }
}
telemetry {
  prometheus_retention_time = "30s",
  disable_hostname = true
}

[cybercharly1988]

The tricky bit which took me a while to work out is that the unauthenticated_metrics_access needs to be within your listener config e.g.

listener "tcp" {
  telemetry {
    unauthenticated_metrics_access = "true"
  }
}
telemetry {
  prometheus_retention_time = "30s",
  disable_hostname = true
}

thanks basically this is the answer, create separate section for every telemetry, i have had wasted almost 5 hours fixing this issue, thanks

[tyriis]

have translated the prom config to a pod monitor

---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: vault
  namespace: secops
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: fluxcd
    app.kubernetes.io/name: vault
spec:
  namespaceSelector:
    matchNames:
      - secops
  selector:
    matchLabels:
      app.kubernetes.io/instance: vault
      app.kubernetes.io/name: vault
      vault-active: "true"
  podMetricsEndpoints:
    - path: /v1/sys/metrics
      params:
        format: ["prometheus"]
      port: http
      relabelings:
        - action: keep
          sourceLabels: ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_port_number"]
          regex: secops;8200

assure your namesapce matches when you copy it ;)

[jsmickey]

Since this post helped with the undocumented telemetry settings, I wanted to share this:

I deployed the Prometheus Helm Chart
Add the following to https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml

This automatically imports the endpoints and sets them as targets in Prometheus.

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      - job_name: 'vault'
        metrics_path: '/v1/sys/metrics'
        params:
          format: ['prometheus']
        scheme: https
        tls_config:
          ca_file: '/etc/prometheus/secrets/my-secret/ca.crt'
          insecure_skip_verify: true
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
          - source_labels:
              [
                __meta_kubernetes_namespace,
                __meta_kubernetes_pod_container_port_number,
              ]
            action: keep
            regex: vault;8200

[tyriis]

@june07 restart is not related to vault, it is the default kubernetes functionality to not restart pods when the config-map or secret changes (they are loaded into the container but most applications read config only start in the lifecycle. To bypass this limitation, have a look at https://github.com/stakater/Reloader.

[zhangzheyu2simple]

$ git branch

• master

[zhangzheyu2simple]

while I can get a json format metrics using curl -X GET "http://localhost:8236/v1/sys/metrics -H "X-Vault-Token: <root_token>"

[tvoran]

Hi @zhangzheyu2simple, I wasn't able to reproduce this bug with your given values file; the telemetry stanza appears correct for enabling prometheus. Since the json format metrics were accessible, it sounds like the config from your helm values isn't making it into the ConfigMap.

I'd suggest double-checking which values are being used in the deployment (helm get values vault), and also checking the deployed config map. If you run kubectl describe configmap vault-config, that should contain the telemetry stanza with the prometheus_retention_time setting.

[cablespaghetti]

#215 Should help you get up and running with Prometheus a little more easily.

[tvoran]

Thanks @cablespaghetti, we'll take a look.

Closing this issue for now, let us know if you run into further issues @zhangzheyu2simple.

[damianfedeczko]

Hello,
I think that your issue is connected to the fact that you are running Vault in HA with 3 instances.
Accordingly to the docs:
"The /v1/sys/metrics endpoint is only accessible on active nodes and automatically disabled on standby nodes. You can enable the /v1/sys/metrics endpoint on standby nodes by enabling unauthenticated metrics access."
https://www.vaultproject.io/docs/configuration/telemetry#prometheus

Your issue would be probably solved by adding unauthenticated_metrics_access = true to your telemetry stanza - it worked for me when deployed HA with 3 Vault instances. When running 1 Vault instance, the directive is not needed.

Really hope that this will help with your issue, cheers!

[one1zero1one]

@tvoran We're seeing a similar behaviour - thought not to open a new issue since this one is quite recently closed.

chart: hashicorp/vault
version: 0.6.0

The relevant bits from the values.yaml:

  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      log_format = "json"
      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      telemetry {
        unauthenticated_metrics_access = true
        prometheus_retention_time = "24h"
        disable_hostname = true
      }
      storage "consul" {
        path = "vault"
        address = "vault-consul-server.vault.svc.cluster.local:8500"
      }

I've checked the configmap and container and the configuration made it ok there:

/ $ cat /tmp/storageconfig.hcl
...
telemetry {
  unauthenticated_metrics_access = true
  prometheus_retention_time = "24h"
  disable_hostname = true
}
...
ps axf
...
    9 vault     0:02 vault server -config=/tmp/storageconfig.hcl
...

However, when trying to scrape with prometheus we get 400 Bad Request

curl http://10.4.80.34:8200/v1/sys/metrics?format=prometheus
prometheus is not enabled

[damianfedeczko]

@one1zero1one can you try `curl -X GET 'http://$YOUR_VAULT_INSTANCE/v1/sys/metrics?format=prometheus' -H "X-Vault-Token:$YOUR_TOKEN" ? Ditch the 8200 in curl.

[damianfedeczko]

@cablespaghetti yeah, the docs are a little bit misleading in this case.

As described here, the unauthenticated_metrics_access telemetry directive has to declared within the listener, just like you did:
https://www.vaultproject.io/docs/configuration/listener/tcp#configuring-unauthenticated-metrics-access
But, when looking for the telemetry configuration docs for Prometheus, you are also instructed to use a telemetry stanza - just not embedded into the listener config. Confusing, should be unified in my opinion.

[one1zero1one]

@cablespaghetti thanks, that solved it.

/ # curl http://10.4.80.40:8200/v1/sys/metrics?format=prometheus
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection 
...

@damianfedeczko thanks - but didn't get a chance to try it as my colleague ran the new config faster than I could check - but I would assume would have worked to use the service and token. Issue was that we aimed for unauthenticated scrape from the get go.

+1 for ultimately having the telemetry stanza unified to avoid confusion.

[damianfedeczko]

@one1zero1one cool, no worries - @cablespaghetti answer nailed it

[mshivanna]

the above solution works. After adding the above configuration, if you are running vault on Prometheus you will have to restart the pods.

[june07]

Heads up for anyone else to come across this... a restart is needed for the settings to take effect. Simple reload was not sufficient. Maybe someone more familiar with the code base can confirm as well.

Was trying to avoid needing to unlock vault again. Ah well.

[ellipsis-me]

The tricky bit which took me a while to work out is that the unauthenticated_metrics_access needs to be within your listener config e.g.

listener "tcp" {
  telemetry {
    unauthenticated_metrics_access = "true"
  }
}
telemetry {
  prometheus_retention_time = "30s",
  disable_hostname = true
}

That worked for me, tks!