weaveworks / grafanalib

Python library for building Grafana dashboards

Getting Started with grafanalib

Do you like Grafana but wish you could version your dashboard configuration? Do you find yourself repeating common patterns? If so, grafanalib is for you.

grafanalib lets you generate Grafana dashboards from simple Python scripts.

How it works

Take a look at the examples directory, e.g. this dashboard will configure a dashboard with a single row, with one QPS graph broken down by status code and another latency graph showing median and 99th percentile latency.

In the code is a fair bit of repetition here, but once you figure out what works for your needs, you can factor that out. See our Weave-specific customizations for inspiration.

You can read the entire grafanlib documentation on readthedocs.io.

Getting started

grafanalib is just a Python package, so:

$ pip install grafanalib

Generate the JSON dashboard like so:

$ curl -o example.dashboard.py https://raw.githubusercontent.com/weaveworks/grafanalib/main/grafanalib/tests/examples/example.dashboard.py
$ generate-dashboard -o frontend.json example.dashboard.py

Support

This library is in its very early stages. We'll probably make changes that break backwards compatibility, although we'll try hard not to.

grafanalib works with Python 3.6 through 3.11.

Developing

If you're working on the project, and need to build from source, it's done as follows:

$ virtualenv .env
$ . ./.env/bin/activate
$ pip install -e .

Configuring Grafana Datasources

This repo used to contain a program gfdatasource for configuring Grafana data sources, but it has been retired since Grafana now has a built-in way to do it. See https://grafana.com/docs/administration/provisioning/#datasources

Community

We currently don't follow a roadmap for grafanalib and both maintainers <https://github.com/weaveworks/grafanalib/blob/main/MAINTAINERS> have recently become somewhat occupied otherwise.

We'd like you to join the grafanalib community! If you would like to help out maintaining grafanalib that would be great. It's a fairly laid-back and straight-forward project. Please talk to us on Slack (see the links below).

We follow the CNCF Code of Conduct.

Getting Help

If you have any questions about, feedback for or problems with grafanalib:

• Read the documentation at https://grafanalib.readthedocs.io
• Invite yourself to the Weave Users Slack.
• Ask a question on the #grafanalib slack channel.
• File an issue.

Your feedback is always welcome!

grafanalib's Issues

Publish documentation on readthedocs

We already have a bit of API documentation. Would be good if that were made available on readthedocs.

RFC: Use "constantly" for enums

Switching to using attrs was a huge help. I'd like to do the same thing for the many, many sets of constants that we have. http://constantly.readthedocs.io/en/latest/ looks like a pretty good library, and has had the advantage of a long period of testing.

e.g.

# Y Axis formats
DURATION_FORMAT = "dtdurations"
NO_FORMAT = "none"
OPS_FORMAT = "ops"
PERCENT_UNIT_FORMAT = "percentunit"
SECONDS_FORMAT = "s"
MILLISECONDS_FORMAT = "ms"
SHORT_FORMAT = "short"
BYTES_FORMAT = "bytes"

which is used as

    yAxes = attr.ib(
        default=attr.Factory(lambda: [YAxis(format=SHORT_FORMAT)] * 2))

would become something like:

from constantly import ValueConstant, Values
class YAxisFormat(Values):
    """Y-axis formats."""
    DURATION = ValueConstant("dtdurations")
    NONE = ValueConstant("none")
    OPS = ValueConstant("ops")
    PERCENT_UNIT = ValueConstant("percentunit")
    SECONDS = ValueConstant("s")
    MILLISECONDS = ValueConstant("ms")
    SHORT = ValueConstant("short")
    BYTES = ValueConstant("bytes")

which would be used as:

    yAxes = attr.ib(
        default=attr.Factory(lambda: [YAxis(format=YAxisFormat.SHORT)] * 2))

We can then write validators that use .iterconstants() or .lookupByName() to validate the constants.

I don't know how this will affect backwards compatibility, or indeed what backwards compatibility goals we should have.

How does one build grafanalib?

I can't tell how to build grafanalib, especially generate-dashboards from the README. How should one do it?

Move PromGraph to core module

Prometheus is a super common use case. Current PromGraph conflates Weave config with using Prometheus.

• Get rid of the hard-coded data source
• Move function to core
• Re-export from _weave using hard-coded data source

Support influxDB datasource?

I would like to ask is there a support for influxdb datasource? And if possible, what would look like Target( expr='<Influxdb data source ?>', legendFormat="1xx", refId='A', ),
Thanks for response.

Cloudwatch datasources not supported due to Target class restrictions

There is no support for Cloudwatch since the Target class has restrictions. If you remove them, you'll be able to make Cloudwatch graphs.

Mutable defaults for many Grafanalib attributes

Lots of the attributes in grafanalib have mutable defaults. (e.g. [] or some other grafanalib value). See http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments for why this is a bad idea.

Instead of using empty list, or grafanalib classes, we should use attr.Factory.

Make targets / documentation for building

Currently assume people developing this are familiar with standard Python packaging set up, but that's not a valid or even fair assumption.

We should have make targets that build virtualenvs & have documentation for the overall process.

Question: How many types of panel does grafanalib support?

The example only shows that grafanalib supports Graph panel, does it also support other types, like singlestat, heatmap?

Thanks.

Jsonnet version?

This isn't a bug, so much as a question for any grafanalib users who might be watching.

How would you feel about a jsonnet version of grafanalib?

Prompted by recent announcement of ksonnet.

Grafana compatibility issue

Hi guys, your lib is awesome, and I want to help with its maintenance, so I've installed it from master branch, but It seems that master is not really working with the latest grafana (4.11, 4.12).
What grafana version is library compatible with? Release 0.12 is working ok, but you've made lots of changes since then.

SingleStat does not support timeFrom

I want use timeFrom change time range that Override relative time
but SingleStat does not support

Question: Prometheus only?

Does this library only work with prometheus? We use Telegraf and collect metrics and events into InfluxDB. Would it work with those?

Thanks in advance!

Make it easier to use grafanalib in a project

There's a bit of a documentation / tooling gap between writing a file as code and getting it into your Kubernetes cluster.

At Weaveworks, we have a fairly complex Makefile that builds a grafana image.

We should either document how that works, or make a higher-level tool that does all of that for you.

Question: Is there any way to specify the size of panels?

Hi everyone, I'm creating dashboards using grafanalib but there is no way to specify size or position in a specific panel. I tried to create my own class with a gridPos object (x and y position and height and width attributes) and I'm able to generate a json with correct gridPos but anyway it doesn't work.

Am I doing something wrong? please help

Grafana 5 compatibility

I'm starting this to track Grafana 5 compatibility issues as we find them, we could break them out into multiple issues later?

• Dashboard uid required for consistent URLs #125
• Rows without titles show a title of "New Row"
• Heatmap Panels

Graph object does not support decimals

There is a use case where the displayed value on a Graph needs to have as specific decimal value set to it but the current implementation does not allow that.

Check for valid version of Python in the Makefile

It checks for various dependencies such as virtualenv and pip, but not whether you have Python 3.4 (I had 2.7).

IndexError: tuple index out of range

$ docker run -ti quay.io/weaveworks/gfdatasource:master-9085ab0 -h
Traceback (most recent call last):
  File "/gfdatasource", line 254, in <module>
    main()
  File "/gfdatasource", line 238, in main
    opts = parser.parse_args(sys.argv[1:])
  File "/gfdatasource", line 156, in parse_args
    global_args += (arg, remaining[0])
IndexError: tuple index out of range

grafanalib-specific linter

Weaveworks has its own monitoring linter that enforces a lot of the things that grafanalib does anyway. It'd be nice to have something for users of grafanalib to run that linted their own configurations.

For bonus points, it should know what's exported by grafanalib so it can check for unrecognised names.

Automate release process

I've got a checklist, but it's still pretty fraught with human error. Would like to just push a button. Probably should use a docker image.

Update docs re data source for new grafana releases

ISTR new grafana releases have better ways of specifying the data source. We should find those and refer to them in the docs, promoting gfdatasource only for older versions.

Automatically calculate "id"

Currently, Graphs have to specify their ID. Ideally, the generator would just populate this uniquely.

Fixing this probably requires a shift in our internals. Rather than returning the "rendered" JSON dict we should return a custom object (e.g. a Graph type) with a toJSON method. When we have such objects, it will be fairly easy to add a post-processing stage to insert ID.

Push gfdatasource to docker hub as well as quay.io

The blog post refers to weaveworks/gfdatasource:latest, but we aren't pushing there any more. We should keep pushing there because:

• it keeps the blog post relevant
• people look for things on DockerHub

CC @errordeveloper

Make generate-dashboard(s) into a module?

First of all, thanks for creating this project! I am wondering though if the generate-dashboard/s scripts could be made into a module (in addition to the scripts) so one could generate dashboards from one's own scripts?

I could contribute a PR if you think it's a good idea.

Parameterise Template class fields

Currently the Template class only exposes certain fields (

grafanalib/grafanalib/core.py

Lines 370 to 409 in 8bbdbc8

	class Template(object):
	"""Template create a new 'variable' for the dashboard, defines the variable
	name, human name, query to fetch the values and the default value.
	:param default: the default value for the variable
	:param dataSource: where to fetch the values for the variable from
	:param label: the variable's human label
	:param name: the variable's name
	:param query: the query users to fetch the valid values of the variable
	"""
	default = attr.ib()
	dataSource = attr.ib()
	label = attr.ib()
	name = attr.ib()
	query = attr.ib()
	def to_json_data(self):
	return {
	'allValue': None,
	'current': {
	'text': self.default,
	'value': self.default,
	'tags': [],
	},
	'datasource': self.dataSource,
	'hide': 0,
	'includeAll': False,
	'label': self.label,
	'multi': False,
	'name': self.name,
	'options': [],
	'query': self.query,
	'refresh': 1,
	'regex': '',
	'sort': 1,
	'tagValuesQuery': None,
	'tagsQuery': None,
	'type': 'query',
	}

) but others are given a default value that may not apply for each use case.
The class should allow for these parameters to be overwritten.

Define `Annotation` type

e.g.

{
    "datasource": "Weave Cloud Events",
    "enable": True,
    "hide": False,
    "iconColor": "rgba(255, 96, 96, 1)",
    "limit": 100,
    "name": "Weave Cloud Events",
    "showIn": 0,
    "type": "alert"
}

Add Text panel definition

Hi!
I was playing around with grafanalib and noticed a Text panel definition is missing, are the plans to add it? Happy to work on a PR too if that's easier

Automatically test examples

It's a bit of a pain to manually test the thing in the docs

Any plans for 1.0.0?

First of all, thanks for this tremendously helpful library.

When do you plan to have a 1.0.0 release?

Specify target option like intervalFactor in PromGraph

Looking at https://github.com/weaveworks/grafanalib/blob/master/grafanalib/prometheus.py#L8

The PromGraph function have a fix call to Target to insert expressions with only expr, legend, refId=refId.

In my case I need to also specify other parameters like intervalFactor.

WDYT about change expressions list of tuple arg by list of dict to allow to specify every Target args ?

If needed we could let both behavior compatible by checking if the list contain tuple or dict.

Something like

    if all(isinstance(expr, dict) for expr in expressions):
        targets = [ 
            G.Target(refId=refId, **args)
            for (args, refId) in zip(expressions, letters)]
    else:
        targets = [ 
            G.Target(expr, legend, refId=refId)
            for ((legend, expr), refId) in zip(expressions, letters)]

Could do the job

Question: Would you consider tagging a new release so we can get hold of the uid functionality?

Current commits post the 0.5.1 release have: version='0.5.0' ... which makes it a little difficult to pull commits via python setup.py that enable the uid functionality (v useful in grafana-5).

So, I was wondering if you'd consider tagging another release 0.5.2, which would make that much more straightforward?

Thanks,
Roland

Use lint utils from tools directory

We should use that!

PROPOSAL: classes refactor

This one is intended to start a discussion, here is my proposal:

All the classes must have to_json_data method, so it's a good idea to factor it out in a parent class.

It can be something like this:

@attr.s
class JsonObject(object):

    def to_json_data(self):
        return dict(
            (a.name, getattr(self, a.name))
            for a in self.__attrs_attrs__ if getattr(self, a.name) is not None)

we'll have one major problem with this approach: it seams that there is no hard rule on grafana json fields naming. some fields are in camelcase, some are in snakecase.
So, we can either keep class field names equal to grafana json names, or introduce some kind of mapping, like this:

@attr.s
class JsonObject(object):
    FIELD_MAPPING = {
        'dataSource': 'datasource'
    }

    def get_field_mapping(self, field):
        return self.FIELD_MAPPING.get(field, field)

    def to_json_data(self):
        return dict(
            (self.get_field_mapping(a.name), getattr(self, a.name))
            for a in self.__attrs_attrs__ if getattr(self, a.name) is not None)

README example doesn't work

When trying to generate the output for the example in the README file I get a NameError: name 'Alert' is not defined error.

generate-dashboard -o frontend.json frontend.dashboard.py
Traceback (most recent call last):
  File "/usr/local/bin/generate-dashboard", line 11, in <module>
    sys.exit(generate_dashboard_script())
  File "/usr/local/lib/python3.4/site-packages/grafanalib/_gen.py", line 108, in generate_dashboard_script
    run_script(generate_dashboard)
  File "/usr/local/lib/python3.4/site-packages/grafanalib/_gen.py", line 98, in run_script
    sys.exit(f(sys.argv[1:]))
  File "/usr/local/lib/python3.4/site-packages/grafanalib/_gen.py", line 85, in generate_dashboard
    dashboard = load_dashboard(opts.dashboard)
  File "/usr/local/lib/python3.4/site-packages/grafanalib/_gen.py", line 18, in load_dashboard
    module = SourceFileLoader("dashboard", path).load_module()
  File "<frozen importlib._bootstrap>", line 539, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1614, in load_module
  File "<frozen importlib._bootstrap>", line 596, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 1220, in load
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/opt/code/frontend.dashboard.py", line 44, in <module>
    alert=Alert(
NameError: name 'Alert' is not defined

System

Python 3.4.6
pip 9.0.1 from /usr/local/lib/python3.4/site-packages (python 3.4)
grafanalib 0.1.2
Linux 7ccfd83caf20 4.9.27-moby #1 SMP Thu May 11 04:01:18 UTC 2017 x86_64 Linux

Small improvement proposal on Legend class

There are four class fields that define what type of values grafana will show in legend

    avg = attr.ib(default=False, validator=instance_of(bool))
    current = attr.ib(default=False, validator=instance_of(bool))
    max = attr.ib(default=False, validator=instance_of(bool))
    min = attr.ib(default=False, validator=instance_of(bool))

But, grafana won't actually show them if you wont set values field to True too.
May be we should automatically set values to True if any of avg, current, max or min are set to True

XAxis Object missing Attributes

What you expected to happen?

When creating an XAxis Object expect more then just the show attribute on the XAxis Object to be translated into JSON.

What happened?

There is no error message in this case. However, if you look at the JSON that is generated, only the show key is set.

"xaxis": {
    "show": true
}, 

I expect the following:

"xaxis": {
    "mode": "series", 
     "name": null, 
     "show": true, 
     "values": [
         "current"
     ]
}, 

How to reproduce it?

• Create a Graph Object
• Give it a xAxis attribute
• Create a XAxis see below

xAxis=XAxis(
    show=True,
    mode="series",
    values=["current"],
),

• Observe generated JSON

"xaxis": {
     "show": true
}, 

I would propose that the XAxis Class should look like this:

@attr.s
class XAxis(object):

    mode = attr.ib(default="time", validator=is_valid_xaxis_mode)
    name = attr.ib(default=None)
    values = attr.ib(default=attr.Factory(list))
    show = attr.ib(validator=instance_of(bool), default=True)

    def to_json_data(self):
        return {
            'show': self.show,
            'mode': self.mode,
            'values': self.values,
            'name' : self.name,
        }

Specifying two Y axes is cumbersome

Lots of graphs have configuration like:

        yAxes=[
          YAxis(format=OPS_FORMAT),
          YAxis(format=SHORT_FORMAT),
        ],

The second Y axis isn't very interesting at all. If it's not interesting, we shouldn't have to configure it.

Support for common module(s) between dashboards

At the moment, dashboards can only import things that are in the current PYTHONPATH. That means if you want to have a module with some common code (e.g. common.py) in the same directory as your dashboards, you need to run something like:

PYTHONPATH=/path/to/dashboards:$PYTHONPATH generate-dashboards -o /path/to/output/foo.json /path/

Probably easiest thing is to add directory of dashboard files to PYTHONPATH (perhaps with a flag).

Relates to #57

Warn when only one Y axis provided

Grafana needs two. If there's only one provided, it will render as an error.

Correctly format README on PyPI

Currently you see raw rST markup. Want to see nice formatting.

Don't use build-tools

grafanalib doesn't benefit much from using Weaveworks' common build-tools, as it doesn't need to generate images (now that gfdatasource is obsolete), and it is not built with Go. We should just stop using it.

Tool for generating grafanalib dashboards from JSON

For people getting started with grafanalib, it would be helpful to have a way of converting JSON dashboards into grafanalib ones.

Since grafanalib doesn't yet support everything grafana does, when the tool encounters an unsupported feature, it should fail with a friendly message encouraging the user to file a bug.

Document the use of Templating

What you expected to happen?

I attempted to use add templating to a dashboard, it is not documented how to use it so I provided these parameters:
https://github.com/aknuds1/grafanalib/blob/master/grafanalib/core.py#L431-L439

        :param default: the default value for the variable
        :param dataSource: where to fetch the values for the variable from
        :param label: the variable's human label
        :param name: the variable's name
        :param query: the query users to fetch the valid values of the variable
        :param allValue: specify a custom all value with regex,
            globs or lucene syntax.
        :param includeAll: Add a special All option whose value includes
            all options.

What happened?

Js error in grafana when trying to import the resulting json dashboard. After some effort I determined that the js error is caued by the absence of some of these values for each template:
https://github.com/aknuds1/grafanalib/blob/master/grafanalib/core.py#L456-L470

    def to_json_data(self):
        return {**_as_dict(self), **{
            'current': {
                'text': self.default,
                'value': self.default,
                'tags': [],
            },
            'hide': 0,
            'multi': False,
            'options': [],
            'refresh': 1,
            'regex': '',
            'sort': 1,
            'tagValuesQuery': None,
            'tagsQuery': None,
            'type': 'query',
        }}

I'm not very familiar with python but that bit of code looks like it should be used as defaults? In any case, when I include those keys/values for each templated variable in my dashboard.py, the resulting json generated does work!

How to reproduce it?

Create any dashboard that uses the templating, i.e.:

  templating=Templating(list=[
    { # namespace
      "default": "",
      "datasource": "prometheus",
      "label": "Namespace",
      "name": "namespace",
      "query": "label_values(kube_pod_info, namespace)",
      "allValue": ".+",
      "includeAll": True,
    },
  ]),

Is this a bug with how templating gets rendered? If not, can you add the proper incantation to use templating in the example dashboard.py?

Require newer release version

Several new attributes has been supported in recent commits. It's not a good practice to install python package from github repo, I hope there's a new release version on Pypi, thanks.

Make all Grafanalib objects immutable

I would like to make all Grafanalib objects immutable, using attrs's frozen setting (see http://www.attrs.org/en/stable/examples.html#immutability).

Pros

• Easier to reason about
• Fewer bugs
• Reduced API support surface (don't have to worry about behaviour when people change things)

Cons

• People might be mutating things in their current dashboards, and this will break those changes.

I welcome input from existing users.

Feature Request: Convert from JSON to python equivalent

I know this is a huge ask, but it would be awesome if this library could convert from json to the python library equivalent. It would save a lot of time from having to manually craft the python grafanalib compatible file.

Automatic refId generator

I'd really like to see one, similarly to now auto_panel_ids() currently work: I just don't care about refIds most of the time, yet omitting them (just like in that minimal example below) causes all sorts of weird problems on grafana side, like in 5.1.3 where sibling graphs with empty refIds render nothing but loading spinner up until query inspector is opened.

dashboard = Dashboard(
        title = 'testing testing 123',
        rows = [
                Row(panels = [
                        Graph(
                                title = 'zomg constants',
                                dataSource = 'prometheus',
                                targets = [
                                        Target(expr = '1'),
                                        Target(expr = '2'),
                                ],
                        ),
                ]),
        ],
).auto_panel_ids()

Example code doesn't work

Just a general comment, the example provided in the readme as well the example code on blog post https://www.weave.works/blog/grafana-dashboards-as-code/ do not work out of the box. As a new user this is very frustrating. I would like to be able to cargo cult the example code and it should work with generate-dashboard command. It may possibly make sense to include a more complete set of examples in an examples directory?

grafanlib didn't work due to "No module named machinery"

Is grafanalib supports python7.2 ?!
grafnalib were installed via pip install grafanalib
when running grafanalib, 'm facing this:
generate-dashboard -o frontend.json frontend.dashboard.py
Traceback (most recent call last):
File "/usr/bin/generate-dashboard", line 9, in
load_entry_point('grafanalib==0.4.0', 'console_scripts', 'generate-dashboard')()
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/usr/lib/python2.7/site-packages/grafanalib/_gen.py", line 7, in
from importlib.machinery import SourceFileLoader
ImportError: No module named machinery

I just exeect the grafnalib to run as described

looks like it didn't worked for py27
pip install py2exe
RuntimeError: This package requires Python 3.3 or later
see also https://stackoverflow.com/questions/24237385/no-module-named-machinery