Dashboard helm chart 7.3.2 no graphics about dashboard

[floreks]

You need to provide logs from all dashboard containers first.

[Alezzzander]

During log collection,i found this line in dashboard-api container:

E0502 11:07:49.844195       1 manager.go:96] Metric client health check failed: the server is currently unable to handle the request (get services kubernetes-dashboard-metrics-scraper). Retrying in 30 seconds.

Can it be the reason?

dashboard-api has these args:

  containers:
    - args:
        - '--namespace=kube-system'
        - '--metrics-scraper-service-name=kubernetes-dashboard-metrics-scraper'
      env:
        - name: CSRF_KEY
          valueFrom:
            secretKeyRef:
              key: private.key
              name: kubernetes-dashboard-csrf
      image: docker.io/kubernetesui/dashboard-api:1.5.0

dashboard-scraper logs are filled with probes:

10.150.23.46 - - [02/May/2024:11:05:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:05:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:06:02.711109       1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:06:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:07:02.747414       1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:07:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:08:02.701078       1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:08:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"

metrics-server logs:

I0502 10:12:39.303482       1 serving.go:374] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0502 10:12:39.861123       1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
I0502 10:12:39.979541       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0502 10:12:39.979591       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0502 10:12:39.979654       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0502 10:12:39.979740       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0502 10:12:39.979763       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0502 10:12:39.979852       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0502 10:12:39.980179       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0502 10:12:39.981162       1 secure_serving.go:213] Serving securely on [::]:10250
I0502 10:12:39.981201       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0502 10:12:40.080042       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0502 10:12:40.080062       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0502 10:12:40.080152       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file

Metrics-server args:

 --secure-port=10250 
 --cert-dir=/tmp 
 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 
 --kubelet-use-node-status-port 
 --metric-resolution=15s 
 --kubelet-insecure-tls 

[floreks]

Ye, for some reason it can't access metrics scraper. Isn't there more detailed error message in the api logs?

[Alezzzander]

Only these :

[floreks]

Check the beginning of logs when the app is being initialized.

[Alezzzander]

I0502 11:19:08.330608       1 main.go:40] "Starting Kubernetes Dashboard API" version="1.5.0"
I0502 11:19:08.331182       1 init.go:47] Using in-cluster config
I0502 11:19:08.364971       1 main.go:118] "Successful initial request to the apiserver" version="v1.29.3-eks-adc7111"
I0502 11:19:08.365756       1 client.go:268] Creating in-cluster Sidecar client
I0502 11:19:08.381661       1 main.go:95] "Listening and serving on" address="0.0.0.0:8000"
2024/05/02 11:19:08 Getting list of all cron jobs in the cluster
2024/05/02 11:19:09 Getting list of all replica sets in the cluster
2024/05/02 11:19:09 Getting list of all replication controllers in the cluster
2024/05/02 11:19:09 Getting list of ingress classes in the cluster
2024/05/02 11:19:09 Getting list of all services in the cluster
2024/05/02 11:19:09 Getting list of all stateful sets in the cluster
2024/05/02 11:19:09 Getting list of all pods in the cluster
2024/05/02 11:19:09 Getting list of all deployments in the cluster
2024/05/02 11:19:09 Getting list of all jobs in the cluster
2024/05/02 11:19:09 Getting list config maps in the namespace 
2024/05/02 11:19:09 Getting list of secrets in &{[]} namespace
2024/05/02 11:19:09 Getting list persistent volumes
2024/05/02 11:19:09 Getting list of RBAC roles
2024/05/02 11:19:09 Getting list of storage classes in the cluster
2024/05/02 11:19:09 Getting list of all roleBindings in the cluster
2024/05/02 11:19:09 Getting list of all roles in the cluster
2024/05/02 11:19:09 Getting list of namespaces
2024/05/02 11:19:09 Getting list of all clusterRoleBindings in the cluster
2024/05/02 11:19:09 Getting list of all deployments in the cluster
2024/05/02 11:19:13 Getting list persistent volumes
2024/05/02 11:19:18 Getting pod metrics
2024/05/02 11:19:39 Getting list of all deployments in the cluster
2024/05/02 11:19:54 Getting list of all deployments in the cluster
2024/05/02 11:19:58 Getting list of namespaces
2024/05/02 11:19:58 Getting list of all cron jobs in the cluster
2024/05/02 11:19:58 Getting list of all pods in the cluster
2024/05/02 11:19:58 Getting list of all replication controllers in the cluster
2024/05/02 11:19:58 Getting list of all deployments in the cluster
2024/05/02 11:19:58 Getting list of all jobs in the cluster
2024/05/02 11:19:58 Getting list of all replica sets in the cluster
2024/05/02 11:19:58 Getting list of all stateful sets in the cluster
2024/05/02 11:19:58 Getting pod metrics
2024/05/02 11:19:59 Getting list of namespaces
2024/05/02 11:19:59 Getting list of all cron jobs in the cluster
2024/05/02 11:19:59 Getting list of all deployments in the cluster
2024/05/02 11:19:59 Getting list of all deployments in the cluster
2024/05/02 11:19:59 Getting list of all jobs in the cluster
2024/05/02 11:19:59 Getting list of all replication controllers in the cluster
2024/05/02 11:19:59 Getting list of all replica sets in the cluster
2024/05/02 11:19:59 Getting list of all pods in the cluster
2024/05/02 11:19:59 Getting list of all stateful sets in the cluster
2024/05/02 11:19:59 Getting pod metrics
2024/05/02 11:20:01 Getting list of all deployments in the cluster
2024/05/02 11:20:02 Getting list of all deployments in the cluster
2024/05/02 11:20:03 Getting list of all jobs in the cluster
2024/05/02 11:20:04 Getting list of all pods in the cluster
2024/05/02 11:20:04 Getting list of all jobs in the cluster
2024/05/02 11:20:04 Getting list of all deployments in the cluster
2024/05/02 11:20:04 Getting list of all cron jobs in the cluster
2024/05/02 11:20:04 Getting list of namespaces
2024/05/02 11:20:04 Getting list of all stateful sets in the cluster
2024/05/02 11:20:04 Getting list of all replication controllers in the cluster
2024/05/02 11:20:04 Getting list of all replica sets in the cluster
2024/05/02 11:20:04 Getting pod metrics
2024/05/02 11:20:04 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all deployments in the cluster
2024/05/02 11:20:06 Getting list of all cron jobs in the cluster
2024/05/02 11:20:06 Getting list of namespaces
2024/05/02 11:20:06 Getting list of all replication controllers in the cluster
2024/05/02 11:20:06 Getting list of all replica sets in the cluster
2024/05/02 11:20:06 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all stateful sets in the cluster
2024/05/02 11:20:06 Getting pod metrics
2024/05/02 11:20:06 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all jobs in the cluster
2024/05/02 11:20:07 Getting list of all deployments in the cluster
2024/05/02 11:20:07 Getting pod metrics
2024/05/02 11:20:09 Getting list of all deployments in the cluster
2024/05/02 11:20:10 Getting pod metrics
2024/05/02 11:20:11 Getting list of namespaces
2024/05/02 11:20:11 Getting list of all pods in the cluster
2024/05/02 11:20:11 Getting pod metrics
2024/05/02 11:20:12 Getting list of all deployments in the cluster
2024/05/02 11:20:13 Getting list of all replica sets in the cluster
2024/05/02 11:20:14 Getting list of namespaces
2024/05/02 11:20:15 Getting list of all replica sets in the cluster
2024/05/02 11:20:17 Getting list of all deployments in the cluster
2024/05/02 11:20:17 Getting list of all deployments in the cluster
2024/05/02 11:20:18 Getting list of namespaces
2024/05/02 11:20:20 Getting details of kubernetes-dashboard-metrics-server deployment in kube-system namespace
2024/05/02 11:20:22 Getting list of all deployments in the cluster
2024/05/02 11:20:24 Getting list of namespaces
2024/05/02 11:20:26 Getting details of kubernetes-dashboard-metrics-server deployment in kube-system namespace
2024/05/02 11:20:26 Getting replication controller kubernetes-dashboard-metrics-server-7c54c778c6 pods in namespace kube-system
2024/05/02 11:20:26 Getting details of kubernetes-dashboard-metrics-server-7c54c778c6 service in kube-system namespace
2024/05/02 11:20:27 Getting list of all deployments in the cluster
2024/05/02 11:20:27 Getting pod metrics
2024/05/02 11:20:28 Getting list of namespaces
2024/05/02 11:20:31 Getting replication controller kubernetes-dashboard-metrics-server-7c54c778c6 pods in namespace kube-system
2024/05/02 11:20:31 Getting details of kubernetes-dashboard-metrics-server-7c54c778c6 service in kube-system namespace
2024/05/02 11:20:31 Getting list of all deployments in the cluster
2024/05/02 11:20:32 Getting pod metrics
2024/05/02 11:20:33 Getting list of namespaces
2024/05/02 11:20:36 Getting list of all deployments in the cluster
2024/05/02 11:20:37 Getting list of all deployments in the cluster
2024/05/02 11:20:38 Getting list of all deployments in the cluster
2024/05/02 11:20:38 Getting list of namespaces
2024/05/02 11:20:43 Getting list of namespaces
2024/05/02 11:20:45 Getting pod metrics
2024/05/02 11:20:48 Getting list of namespaces
2024/05/02 11:20:53 Getting list of namespaces
2024/05/02 11:20:58 Getting list of namespaces
2024/05/02 11:21:03 Getting list of namespaces
2024/05/02 11:21:08 Getting list of namespaces
2024/05/02 11:21:13 Getting list of namespaces
E0502 11:21:18.808202       1 manager.go:96] Metric client health check failed: the server is currently unable to handle the request (get services kubernetes-dashboard-metrics-scraper). Retrying in 30 seconds.
2024/05/02 11:21:18 Getting list of namespaces
2024/05/02 11:21:23 Getting list of namespaces
2024/05/02 11:21:28 Getting list of namespaces

Here is the log from the beginning

[viram99]

I have used the same configuration and observed similar logs.
I'm encountering the same issue where no graphics are displayed on the Kubernetes dashboard.

[floreks]

You could check if this helps: #8835 (comment)

[viram99]

I have tried adding below args but it still does not work for me.
args:

• --metrics-provider=none
• --sidecar-host=kubernetes-dashboard-metrics-scraper.kubernetes-dashboard

This is the API response for request path /api/v1/pod/ that I have received. any insights to why cumulativeMetrics and metrics field are set to empty and null respectively
{ "listMeta": { "totalItems": 2 }, "cumulativeMetrics": [], "status": { "running": 2, "pending": 0, "failed": 0, "succeeded": 0, "terminating": 0 }, "pods": [{...other pod fields, "metrics":"null" }], "errors": [] }

[floreks]

If you set metrics provider to none you are effectively disabling metrics :). Do not override it or use 'sidecar' as value.

[sigfigsteve]

It needs a protocol and port for the sidecar-host
--sidecar-host=http://kubernetes-dashboard-metrics-scraper.kubernetes-dashboard:8000

[floreks]

@viram99 any progress here?

[floreks]

/close

[k8s-ci-robot]

@floreks: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.