[bug] node dashboard shows no values about grafana-dashboards-kubernetes

[chuegel]

I removed all CRDs and re-deployed the stack again...now the metrics are visible.
Thanks

[dotdc]

Hi @chuegel,
Did you try without the relabelings first ? This should only be used if it doesn't work out of the box.
Can you share your Kubernetes distribution, Kubernetes version and the entire values configuration for kube-prometheus-stack ?

[chuegel]

Hi @dotdc , thanks for your reply.
Yes, I tried that first but the metrics didn't show up.

kubectl version -o yaml
clientVersion:
  buildDate: "2023-06-14T09:53:42Z"
  compiler: gc
  gitCommit: 25b4e43193bcda6c7328a6d147b1fb73a33f1598
  gitTreeState: clean
  gitVersion: v1.27.3
  goVersion: go1.20.5
  major: "1"
  minor: "27"
  platform: linux/amd64
kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "2023-06-14T22:02:13Z"
  compiler: gc
  gitCommit: 25b4e43193bcda6c7328a6d147b1fb73a33f1598
  gitTreeState: clean
  gitVersion: v1.27.3+rke2r1
  goVersion: go1.20.5 X:boringcrypto
  major: "1"
  minor: "27"
  platform: linux/amd64

Here the value configuration:

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: kube-prometheus-stack
spec:
  interval: 1h
  chart:
    spec:
      version: "58.x"
      chart: kube-prometheus-stack
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
      interval: 1h
  install:
    crds: Create
  upgrade:
    crds: CreateReplace
  driftDetection:
    mode: enabled
    ignore:
      # Ignore "validated" annotation which is not inserted during install
      - paths: [ "/metadata/annotations/prometheus-operator-validated" ]
        target:
          kind: PrometheusRule
  valuesFrom:
  - kind: ConfigMap
    name: flux-kube-state-metrics-config
    valuesKey: kube-state-metrics-config.yaml
  # https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml
  values:
    alertmanager:
      enabled: false
    prometheus-node-exporter:
      prometheus:
        monitor:
          relabelings:
          - action: replace
            sourceLabels: [__meta_kubernetes_pod_node_name]
            targetLabel: nodename
    prometheus:
      service:
        labels:
          operated-prometheus: "true"
      prometheusSpec:
        retention: 24h
        resources:
          requests:
            cpu: 200m
            memory: 200Mi
        podMonitorNamespaceSelector: { }
        podMonitorSelector:
          matchLabels:
            app.kubernetes.io/component: monitoring
    grafana:
      defaultDashboardsEnabled: false
      adminPassword: flux
      dashboardProviders:
        dashboardproviders.yaml:
          apiVersion: 1
          providers:
          - name: 'grafana-dashboards-kubernetes'
            orgId: 1
            folder: 'Kubernetes'
            type: file
            disableDeletion: true
            editable: true
            options:
              path: /var/lib/grafana/dashboards/grafana-dashboards-kubernetes
      dashboards:
        grafana-dashboards-kubernetes:
         k8s-system-api-server:
           url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-api-server.json
           token: ''
         k8s-system-coredns:
           url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-coredns.json
           token: ''
         k8s-views-global:
           url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-global.json
           token: ''
         k8s-views-namespaces:
           url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-namespaces.json
           token: ''
         k8s-views-nodes:
           url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json
           token: ''
         k8s-views-pods:
           url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-pods.json
           token: ''

[dotdc]

All the other dashboards are working correct ?

I think you can remove :

    prometheus-node-exporter:
      prometheus:
        monitor:
          relabelings:
          - action: replace
            sourceLabels: [__meta_kubernetes_pod_node_name]
            targetLabel: nodename

Also, can you try to remove podMonitorNamespaceSelector and podMonitorSelector in prometheusSpec and replace them with :

            podMonitorSelectorNilUsesHelmValues: false
            serviceMonitorSelectorNilUsesHelmValues: false

Memory requests for Prometheus might be a bit low, do you have any pod restart or errors?

If nothing works, can you check what is the output of each variable from the node dashboard ?

[chuegel]

All the other dashboards are working correct ?

All, except Nodes, Pods..there are some values in Namespaces and some in Global also missing:

I think you can remove :

    prometheus-node-exporter:
      prometheus:
        monitor:
          relabelings:
          - action: replace
            sourceLabels: [__meta_kubernetes_pod_node_name]
            targetLabel: nodename

done

Also, can you try to remove podMonitorNamespaceSelector and podMonitorSelector in prometheusSpec and replace them with :

            podMonitorSelectorNilUsesHelmValues: false
            serviceMonitorSelectorNilUsesHelmValues: false

Changed that but no difference

Memory requests for Prometheus might be a bit low, do you have any pod restart or errors?

Pods are fine no restarts

If nothing works, can you check what is the output of each variable from the node dashboard ?

For example in the Node dashboard the metric kube_node_info doesn't seem to return any values:

[wigarddev]

I fixed this by renaming label filter nodename to node:

[chuegel]

@wigarddev

when I do that I get: