Dashboard to big for ConfigMap wrapper about kube-prometheus

[brancz]

@eyalzek they are different problems. The configmap generator was there to bin-pack dashboards into configmaps, which was later migrated from python+bash scripts to jsonnet. The problem here is that a single dashboard is larger than the configmap size limit, which no bin-packing can solve as the item is larger than the bin to begin with :)

[afirth]

@michaelpietzsch @metalmatze: Since @aslafy-z 's PR kiwigrid/k8s-sidecar#16 was merged, this works great in the sidecar and is already supported by the helm chart. I believe the sidecar only supports one file/CM though.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-dashboard
  namespace: prometheus-operator
  labels:
    grafana_dashboard: "1"
data:
  # fetch https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/grafana/dashboards/nginx.json with the grafana sidecar
  nginx-ingress.json.url: https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/grafana/dashboards/nginx.json

[metalmatze]

Do you combine all dashboards into one ConfigMap? We actually improve that in brancz/kubernetes-grafana by having one ConfigMap per dashboard.

[michaelpietzsch]

Hi Matthias, I am using the latest version. The system is currently generating one ConfigMap per dashboard. Chances are the config map solution is limited due to the etcd limits. This might be a more fundamental issue when something between config map and persistent storage is needed regards Michael Am 22.10.2018 um 14:37 schrieb Matthias Loibl <[email protected]<mailto:[email protected]>>: Do you combine all dashboards into one ConfigMap? We actually improve that in brancz/kubernetes-grafana<https://github.com/brancz/kubernetes-grafana/> by having one ConfigMap per dashboard. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://github.com/coreos/prometheus-operator/issues/2033#issuecomment-431822931>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKwi_-upi2_j1Uh0shytcJl1sbDXV3pQks5unbwbgaJpZM4XwWg6>.

[metalmatze]

Not sure how we can work around the size limits of ConfigMaps to be honest... 😕
Maybe you can try to improve the dashboard so it also reduces its size?

[michaelpietzsch]

Well there could be the possibility to create some sort of additional shell script which uses a POST request towards the Grafana API to push the dashboard into the system. http://docs.grafana.org/http_api/dashboard/ But its far from ideal and cumbersome. :( Regards Michael Am 23.10.2018 um 13:34 schrieb Matthias Loibl <[email protected]<mailto:[email protected]>>: Not sure how we can work around the size limits of ConfigMaps to be honest... 😕 Maybe you can try to improve the dashboard so it also reduces its size? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://github.com/coreos/prometheus-operator/issues/2033#issuecomment-432210204>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKwi_1uvPPRidB3CZlca46d5-rfBMP7Dks5unv7BgaJpZM4XwWg6>.

[metalmatze]

Yes, a sidecar like this was used in the past by kube-prometheus too:
https://github.com/coreos/prometheus-operator/tree/release-0.19/contrib/grafana-watcher

Otherwise you could extend the Grafana image with your dashboard.

Either way, I don't think there's a way of using the ConfigMaps approach if the dashboard is that big.

[brancz]

Do you know which dashboard is the one that's causing this? A dashboard larger than 1Mb sounds like it would be rather bloated in the first place, improving that sounds like a good idea either way. Otherwise you could maybe look into other volume mounts like a git volume.

[eyalzek]

I don't really understand, as per:
prometheus-operator/prometheus-operator#535
https://github.com/eedugon/grafana-dashboards-configmap-generator

there was some kind of a workaround integrated into the repo, which was merged:
https://github.com/coreos/prometheus-operator/pull/567/files

but it was removed from the repo for some reason? Is there no workaround for this problem using the latest version?

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[brancz]

Closing due to inactivity.