Evaluate used memory as "Total - MemFree - Buffers - Cached" and display stacked Used/Buffers/Cached/Free about kube-prometheus

[rutsky]

BTW, I wrote following bash script for wrapping saved JSONs with dashboards with code you mentioned above:

#!/bin/bash -eu

if [ "$#" -ne 1 ]; then
    echo "Usage: $0 path-to-dashboard.json"
    exit 1
fi

json=$1
temp=$(tempfile -m 0644)

cat >> $temp <<EOF
{
  "dashboard":
EOF

cat $json >> $temp

cat >> $temp <<EOF
,
  "inputs": [
    {
      "name": "DS_PROMETHEUS",
      "pluginId": "prometheus",
      "type": "datasource",
      "value": "prometheus"
    }
  ],
  "overwrite": true
}
EOF

mv $temp $json

[brancz]

Great! Thanks for noticing! Do you want to open a PR for this?

[rutsky]

@brancz

Do you want to open a PR for this?

I'm trying, but looks like there are issues with updating dashboard manifests.

What is intended way of updating manifests in https://github.com/coreos/kube-prometheus/tree/master/assets/grafana ?

I though that it should be:

1. edit dashboard in live Grafana,
2. click "export",
3. update assets/grafana/ manifests with exported jsons.
4. run https://github.com/coreos/kube-prometheus/blob/master/hack/scripts/generate-configmaps.sh and update configman in Kubernetes

But trying to save even unedited dashboard gives me some error about invalid alert specification --- looks like current dashboards were created in significantly older version of Grafana, that is being installed currently.

[brancz]

Unfortunately the API expects a slightly different json from what is exported via the UI. For the dashboards currently available that is:

{
    "dashboard": <here goes the exported json>,
        "inputs": [                                                                                                                                                                                                     
        {                                                                                                                                                                                                           
            "name": "DS_PROMETHEUS",                                                                                                                                                                                
            "pluginId": "prometheus",                                                                                                                                                                               
            "type": "datasource",                                                                                                                                                                                   
            "value": "prometheus"                                                                                                                                                                                   
        }                                                                                                                                                                                                           
    ],                                                                                                                                                                                                              
    "overwrite": true
}

We want to tackle this problem, but have not gotten around doing so. It will most likely be part of prometheus-operator/prometheus-operator#115. It should also be possible to just add the chats as targets into the existing json, that would make it easier to review.

[rutsky]

@brancz wrapping dashboard definition provided code works, thanks!

It should also be possible to just add the chats as targets into the existing json, that would make it easier to review.

I agree that reviewing only minimal changes to the current dashboard specification is more convenient, however it seems it's not easy to generate such minimal changes.
Please take a look how changes looks like if I do them in live Grafana editor: #55

Currently dashboard specification contains lots of dummy/default values, which means that current contents of those json values have lots of unneeded information generated by some generator (e.g. Grafana live editor).
It's not like all contents in current jsons were thoughtfully/carefully written by developers, so there is no point in trying to edit those values by hand (it will be complicated and error prone, e.g. current alert specification in them is incorrect in newer Grafana versions).

Maybe we can somehow normalize/cleanup file contents of dashboard description?
Or just ignore all changes, and lets use version generated by last editor?

Having only semantic content for dashboard descriptions would be nice, but I don't see how this can be achieved with current Grafana editor.

[brancz]

I totally agree, thanks for taking your time on this! Then I would suggest that as long as we're using the bare json, we just use what grafana spits out. Would your PR be ok to review in that case?

A possible solution I can see: We have looked into grafanalib by weaveworks, to be a potential solution for generating the grafana json in a reviewable manner. Do you have an opinion on that?

[rutsky]

For the reference, if I try to "save" currently deployed "all-nodes" dashboard I get following error in Grafana log:

 2017-03-02T13:18:05.135831091Z t=2017-03-02T13:18:05+0000 lvl=eror msg="Invalid alert data. Cannot save dashboard" logger=context userId=1 orgId=1 uname=admin error="Failed to extract alert.Reason: Could not parse frequency "
2017-03-02T13:18:05.138407610Z t=2017-03-02T13:18:05+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db/ status=500 remote_addr=10.233.77.1 time_ms=4ns size=55 

If I open any graph on the dashboard I see that some dummy Alert is defined there:

which is not what intended, AFAIK.

Removing such alert specification from all graphs fixes this error during saving.

[rutsky]

grafanalib by weaveworks looks interesting indeed, as I understand its designed specifically for versioning dashboard specifications.

Looks like using grafanalib will allow to store versioned only semantic content of dashboard, which is great.
From the other side looks like there no way easily edit such dashboards with live preview --- currently I find it very convenient (and this one of the cool features of Grafana) that I am able to do live edits of dashboards with immediate preview.

[brancz]

Agreed. In an ideal world configmaps would be reloaded as soon as we replace them and we would have the same experience, but until we have that we'll have to live like this.

If you can get a workable json together with the added charts/lines that's all we need to do as of right now for that PR, what do you think?

[rutsky]

@brancz done: #56