owid / owid-grapher

A platform for creating interactive data visualizations

owid-grapher

The monorepo we use at Our World in Data to create and publish embeddable, interactive visualizations like this one:

✋ Disclaimer

This repo is currently not well-designed for reuse as a visualization library, nor for reproducing the full production environment we have at Our World in Data, as our tools are tightly coupled with our database structure.

We're gradually making steps towards making our work more reusable, however we still prioritize needs specific to our project that can be at odds with making our tools reusable.

You are still very welcome to reuse and adapt any of our code for your own purposes, and we welcome contributions!

🏎 Quick start

To quickly get a version of the site running for developing Grapher features, we recommend following the local development setup guide.

Additional setup options are also available for other use cases.

🗂 Overview

Multiple projects are maintained in this repo:

Grapher

A client-side interactive data visualization library used by almost every chart on Our World in Data.

All grapher data is stored in a MySQL database that contains both JSON configuration objects for individual charts as well as the data values that they ingest.

The Grapher project is built with Lerna and tsup.

Explorer

A Grapher-based tool that creates more complex data visualization user interfaces.

Each explorer can be configured via a panel in the admin client. Their config files are stored in a separate repository.

Grapher Admin

• A client-side project that provides a user interface for configuring graphers, explorers, and managing and uploading data.

• A server-side project that manages the MySQL database used by graphers.

Baker

A PM2 project that builds a static copy of the Our World in Data website by merging the content authored in Gdocs with the grapher charts created in Grapher Admin.

Site

The React code for rendering our content in pages, used by the Grapher Admin and Baker.

Tooling

Much of our code is based around reactive programming using React and Mobx.

All code is written in TypeScript.

If you want to enable pre-commit hooks, run yarn husky.

Visual Studio Code is recommended for autocompletion and other awesome editor analysis features enabled by static typing.

Why did we start this project?

The following is an excerpt explaining the origin of this repo and what the alternatives tried were (source: Max Roser's Reddit AMA on Oct 17, 2017)

We built the Grapher because there is no similar external tool available. Datawrapper, Tableau, Plotly, various libraries based on d3 are out there but nothing is similar to what the Grapher does for our project.

Before we developed this tool, we built interactive web visualizations by hand through a difficult process of preparing individual spreadsheets of data and then writing custom HTML and JavaScript code to process the contents for each individual visualization. That was pretty painful and it took me hours sometimes to built a chart.

The owid-grapher solves this problem by using a single visualization codebase and crucially a single database into which all of our data is placed. Once the data has been imported, the process of creating a visualization is reduced to simply choosing what kind of visualization is needed and then selecting the relevant variables in the Grapher user interface. The result may then be customized, and is published to the web with the press of a button.

Using our own system has very important advantages:

• Integration with our global development database: Our database of global development metrics is integrated into our visualization tool so that when we add and update empirical data the visualizations are all updated. (In contrast to this, a pre-existing tool would make the exploration of a database impossible and would require the preparation of each dataset separately for each visualisation.)
• Flexibility: We can use automation to change our entire system all at once. For example, if we decide we want to use a different source referencing style, we could easily update this across hundreds of charts. This makes it possible to scale our publication and to sustainably improve our work without starting from scratch at each round.
• Risk mitigation: We hope(!) that Our World in Data is a long-term project and we want the visualizations we produce to continue to be useful and available years from now. An external web service may be shut down or change for reasons we cannot control. We have had this experience in the past and learned our lesson from it.
• Keeping everything up-to-date: Because we want to be a useful resource for some time we make sure that we have a technology in place that allows us to keep all of our work up-to-date without starting from scratch each time. We have our global development database directly integrated in the Grapher and as soon as new data becomes available (for example from a UN agency) we can run a script that pulls in that data and updates all the visualizations that present that data.

---

Cross-browser testing provided by BrowserStack

Client-side bug tracking provided by

owid-grapher's Issues

Error on dataset import page

Hi,
I get this error when going to the "import new data" menu:

Next exception 'ErrorException' with message 'Trying to get property of non-object (View: /var/www/owid-grapher/resources/views/import/index.blade.php)' in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0c104b0ca63521.php:132

I also notice that there is no categories, and no way to add categories.

I followed the installation instructions in the README.md file. Am i missing something?

Error when editing dataset variable.

When editing and changing a dataset variable we get the following error:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'updated_at' in 'field list' (SQL: updatedata_valuessetvalue= 54.84,updated_at= 2016-08-19 06:20:22 whereid= 3413091)

Dual-point interactive timeline on maps

Something we've wanted to do for a while is add interactive timelines to each of the chart types which allow you to specify both a start and end year, then show you the chart for that period.

We currently have this only for scatterplots: https://ourworldindata.org/grapher/life-expectancy-vs-health-expenditure

For choropleth maps, in multi-year mode it would color the map according to the change in values between the two selected years.

Some thought will need to be put into how the legend and color selection works for this!

I've never been suuper happy with the usability of the dual-point interface on scatterplots (particularly switching between single and multi year modes). If we can think of improvements to it that would also be good.

Maybe: add metadata to chart pages for new google data previews

Google recently announced that it will show tabular data in search results if you provide the metadata for it: https://www.blog.google/outreach-initiatives/google-news-initiative/making-it-easier-discover-data-search/

This would be particularly useful for when we make public dataset pages later, but I think we could potentially also go ahead and add it to the static html build for grapher charts now.

Remove legacy metadata columns from variables table

We chatted with @mispy about this—some columns in the variables table were originally used to store metadata, but due to many different metadata formats, originalMetadata is now used to store all metadata as JSON.

Some of these columns are: unit, shortUnit, code, coverage, timespan. Ideally, if they are useful and cannot be derived from anything else, they should be moved to originalMetadata as JSON fields.

Use the stacked area hover behavior for other charts as well

Stacked area hover tooltips look like this:

Hovering over the chart creates a vertical line at a given x value and then shows the y values for each series in descending order.

I quite like this because it lets you easily directly compare all the different values for a year. It occurs to me that there's no reason this behavior couldn't apply just the same to Line and Stacked Bar charts as well-- the only difference would be that it wouldn't display a total in the former case.

Slope chart data transform should be implemented like other charts

The slope chart transform code currently uses this Observations thing I was experimenting with. I ended up deciding not to use that and process data in a simpler way; it should work directly from DimensionWithData.

Origin url field should accept non-ourworldindata.org urls

For core-econ people

Indicate the exact position of a hovered map item on the legend

Like these maps: https://flourish.studio/2018/04/25/london-ward-maps/

We can know exactly where the value should be positionally, so we might as well be precise

Slope charts should have region legends like scatterplots

This one should be reasonably straightforward.

We want to take the little legend from the scatterplot chart type, e.g. https://ourworldindata.org/grapher/urbanization-vs-gdp

And put it on the "slope chart" chart type, e.g. https://ourworldindata.org/grapher/access-to-a-financial-account-or-services

---

The relevant files to look at:

https://github.com/owid/owid-grapher/blob/master/js/charts/ScatterPlot.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/ScatterColorLegend.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/SlopeChart.tsx

Improvements to the label placement algorithm for line charts

A while ago I implemented a positional labelling system based on some of the charts in the book "The Functional Art". It looks like this:

This HeightedLegend algorithm works by placing all the labels at the height of the line, finding all overlaps, and then shifting all overlaps either down or up depending on which direction has available space. In most cases this works well and finds an acceptable solution.

However, there are occasional edge cases like this where it cannot find a solution: https://ourworldindata.org/grapher/mispy-overlap-test

This is because the algorithm always moves all overlapping labels either down or up, it cannot do both simultaneously because it would then need to ensure that it did not create more overlaps in the process of repositioning.

Label placement is a complex problem in computer science, but since this particular problem is 1-dimensional (only the height varies) the algorithm does not need to be super optimized and so I think it should be possible to find a better solution that resolves this edge case (and also has improved labelling behavior in the case when there are clusters in the center).

Port the country name standardizer to JavaScript

Aibek wrote this useful tool for standardizing country names between different formats, a common problem in working with economics datasets: https://github.com/owid/owid-grapher/tree/master/country_name_tool

This is currently server-side code, but we'd like to do most of it client-side. It should be possible to preload the necessary entity data and process the CSV entirely in the browser.

I think the only server-side step should be the remembering of choices in the case when manual resolution is needed.

Once ported to client-side JS, we could then think about moving the standardizer out of the admin and making it a standalone public tool.

Generate a world map on the country standardizer tool

From Max:

Not a necessary feature at all but a little idea which makes the tool look nice and which is relatively easy to set up with what we have built already for Our World in Data: It should be possible that when someone uploads a dataset into the country standardizer that the tool produces a world map with that data. That way people would already have a way to explore the data and they can save world map charts of it.

---

Feature to add in Country Tool: http://localhost:3030/admin/standardize
This is similar to the preview shown in the Variables page: http://localhost:3030/admin/variables

Discrete bar gradient color scheme should show the same color for same values

e.g. https://ourworldindata.org/grapher/mountain-green-cover-index is currently weird

Split the frontend code from owid-grapher into a standalone visualization library

Often people ask what we use to make interactive charts. I can point them at this repo but they can't really just go and adapt it for their own purposes, since everything is tied together as a single web app.

I would like to extract the actual data visualization code from this repository and make it a separate npm dependency that people can easily reuse. Something like nvd3, which we used to depend on ourselves until we found it too limited.

I think the grapher code has some interesting properties that could make it a useful addition to this space, particularly in comparison to older stuff like nvd3:

• Fully responsive, including SVG text wrapping and label overlap detection
• Local SVG and PNG exports with text annotation
• Universal rendering: rare datavis library that can run entirely under node
• Nice Mobx interface for efficient state mutation (chart.title = "foo")
• Designed to integrate into other automated systems (like the chart editor)
• We have the funding and incentive to maintain it for 3+ years

I think we could present it as being a "zero configuration" kind of interface, since that's what it's sort of already designed to be: you give it some data and it infers an interactive chart from it. While some customization options would of course be available on the chart itself, the ideal is that you customize it by annotating the data to give the grapher more information to work with.

We don't want to pursue the kind of flexibility of a system like Vega. The grapher will always do a limited set of things, but we want it to be the best option for doing the things that it can do.

To allow for easy prototyping, it should come in both the developer-focused, npm variant, and also a CDN script tag bundle you can easily include and play with. Also it should have a cool name!

Indexing of static visualizations in the grapher

The grapher has the nice property that the code can keep close track of all the interactive visualizations, make them searchable and show the data they are sourced from, etc. It would be neat if we could also keep track of visualizations produced outside the grapher in the same way, at least to some extent.

The most minimal version would simply be allowing images to be uploaded in the admin with a title, which are published to an image host somewhere. We could then make the images searchable by that title.

Slope charts should allow highlighting and searching entities like scatterplots

We want to take the little legend from the scatterplot chart type, e.g. https://ourworldindata.org/grapher/urbanization-vs-gdp

And put it on the "slope chart" chart type, e.g. https://ourworldindata.org/grapher/access-to-a-financial-account-or-services

Performance is an important consideration here; we want to try to keep the number of SVG elements and updates to them to a minimum.

---

The relevant files to look at:

https://github.com/owid/owid-grapher/blob/master/js/charts/ScatterPlot.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/ScatterColorLegend.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/SlopeChart.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/ControlsFooter.tsx

Public charts index page

We have a lot of visualizations now; let's make it easier for people to find the ones they are looking for.

The minimal version of this would be a page at https://ourworldindata.org/charts or similar which lists the titles of all grapher visualizations and links through to them, with a client-side search function similar to what we use on the admin chart index.

A later version might also include static visualizations or make use of image previews.

Technical considerations:

• It's a page in the Wordpress domain but needs to access the grapher db directly
  • So we want to merge the owid-grapher and owid-theme builds for this
• The page needs to be updated whenever a chart is published or unpublished
• Client-side search is acceptable for now (just a few thousand strings) but if we incorporate more information later we will probably then need an API service of some kind

no import page

With c55431d, I see an empty panel when clicking on "Import New Data". The javascript console says TypeError: props.sourceTemplate is null at localhost:8000/import, line 105, which says new Apps.View.ImportView(importerData).

Error importing new data

The images below is a screenshot of the errors I got when I tried to import new data into the application. Could you help explain what the problem might be. Thanks

End-to-end TypeScript api typing

Would be neat to be able to validate API calls at compile time. Using something like https://github.com/rawrmaan/restyped perhaps

Admin variable previews should pick chart based on kind of data

At the moment it's always maps which doesn't work for non-geographic variables. Should pick like:

Mappable entities => map
Single-year non-mappable data => discrete bar chart
Multi-year non-mappable data => line chart

Allow multiple regression lines for scatterplots

This is a nice straightforward feature request from the core-econ team. Some editor UI, small changes to code and possibly a migration to restructure existing chart configuration slightly should be all that is needed.

Data downloads should exclude data that can't be used by the chart

From Max:

If you download the data behind this chart https://ourworldindata.org/grapher/gini-index-around-2015-1990-2015-countries-vs-gini-index-around-1990-1990-2015-countries

it gives you the population figures for all countries in all years for the last two centuries.

Let’s make a note to change this!

Ideally, the extra data shouldn't even be served in the JSON chart data since it can't be used in any way?

Automate daily upload of database exports to files.ourworldindata.org

So contributors can always have the most recent schema+data

Highlight published datasets in UI (rather than unpublished ones)

Would look nicer I think. Maybe? I'll think about it.

Node scripts with calls to shell commands should fail fast

Currently we use shell.exec from shelljs: https://github.com/shelljs/shelljs

However, shell.exec doesn't throw an exception on a non-zero exit code, and so things can fail in weird ways. We should replace these execs with stricter ones and (importantly) make sure there's no code that implicitly relies on the existing behavior to work.

Is there a better node shell library, maybe one that uses Promises so we can use async/await too?

Don't allow selecting countries without renderable data on scatterplots or slope charts

This is a bug that has been around for a while, but you can see it on e.g. https://ourworldindata.org/grapher/number-of-child-deaths-slope-chart Where it's possible to select Afghanistan but it doesn't appear on the slope chart.

This is because the data selector looks directly at the data to determine what entities are available without reference to the chart type, so it doesn't know whether it's renderable or not. Determining this list of entities should be the responsibility of the chart transform, which has that information.

Not sure what the best way to get that info from the transform to the selector UI is... maybe expose it via an interface on activeTransform?

Work out how migrations will work under full node (typeorm?)

Logo display

When uploading a logo, where should it display? It is not displaying on the graph.

Tracking embed locations for each chart

We have the ability to set an origin url for each chart in the grapher so that the chart can link back to the text. However, this requires manual work and people don't always remember to do it. It would be neat if the code could automatically know which of our pages a chart is embedded.

I think the way this would work is that each time a post is saved, the code goes through the html and picks out any embeds, then updates a database table indicating the relationship. If it is the first such relationship, and the chart has no canonical origin url, it also updates the chart setting the origin url to that page.

This one is a little tricky because of the degree of integration between articles and charts it requires. Probably something to do in conjunction with replacing the Wordpress editor.

Error on import new data page

I get this error when I open the page (import new data):

local.ERROR: ErrorException: Trying to get property of non-object in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0ca63521.php:137

and

Next ErrorException: Trying to get property of non-object (View: /var/www/owid-grapher/resources/views/import/index.blade.php) in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0ca63521.php:137

Looking a little bit I think I got it fixed using the August 5 schema version "Add dataset (sub) categories to schema dump" and updating the "logos" table with the one you have in the last update (Dec 23, 2016)

--
-- Table structure for table `logos`
--

DROP TABLE IF EXISTS `logos`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `logos` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(100) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `svg` text NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

Now apparently it works and loads everything, but I have not had time to prove it yet. I hope it helps

Missing files in the zip archive

I downloaded the zip archive. But by using it, I notice that some files are missing.
Example of the file /vendor/autoload.php

Single-year stacked bar charts

From Hannah:

Max says that Wendy from Core-Econ writes:

"Stacked Bar Charts
When creating bar graphs, the bars are coded to be horizontal. (e.g. see https://ourworldindata.org/grapher/wages-relative-to-the-price-of-energy) Nonetheless, it would be great to have the option of vertical bars. – Non-priority

There is no option for stacked bar graph. Therefore, we cannot input e.g. the following graph: https://core-econ.org/the-economy/book/text/05.html#figure-5-16
Non Priority"

This is something that would be great for us to have also. Sherin amazingly made us stacked bar charts by year (with year on the x-axis) (e.g. https://ourworldindata.org/grapher/deaths-from-natural-disasters-by-type). These are incredibly useful.

What would also be useful is to have stacked bar charts where time is not on the y-axis. In other words, single-year views with countries or other entries as the bars. For example, you could have a country for each stacked bar (you can add and remove countries as we can with discrete bar charts already) with the stacked bar showing meat consumption by type, or energy production by source, or causes of death. This lets you directly compare a number of countries in a given year.

E.g (but maybe best to turn it 90 degrees so the countries are on the x-axis?):

The other possibility Esteban previously raised as helpful here is whether it's easy to have multiple bars (side-by-side) for a given country in a given year. For example, for each country having male and female bars next to each other (with stacks of how they spend their time, or stacks of what they die from). I don't know if this one is as easily doable, but one possibility for the future. Esteban would be happy!

E.g.

Thanks guys!

New chart type: stacked bar chart

Hannah writes:

We have just been discussing which new entries (and updates) which would be good to try to get some material out on in the near future. There are quite a few for which stacked bar charts would be really helpful.

For example, these natural catastrophe ones: https://ourworldindata.org/natural-catastrophes/#decadal-deaths-and-death-rates

For a new causes of death entry, something like this with deaths by age:

---

To implement this, start by copying the Discrete Bar chart type and go from there:

https://github.com/owid/owid-grapher/blob/master/js/charts/DiscreteBarTransform.ts
https://github.com/owid/owid-grapher/blob/master/js/charts/DiscreteBarChart.tsx

These are the relevant hooks for adding new chart types:

https://github.com/owid/owid-grapher/blob/master/js/charts/ChartConfig.tsx#L321
https://github.com/owid/owid-grapher/blob/master/js/charts/ChartTab.tsx#L34
https://github.com/owid/owid-grapher/blob/master/js/admin/EditorBasicTab.tsx#L103

Dual-point interactive timeline on line charts

Like with scatterplots: https://ourworldindata.org/grapher/life-expectancy-vs-health-expenditure

For single-year mode, line charts could become bar charts.

I think this one should be reasonably straightforward. Backwards compatibility may have some interesting edge cases.

Nice REPL for exploring the code and database

Always find myself missing rails console from my Ruby days. Can we get something similar with node? Just needs to open a basic REPL with the global scope already populated with most of the project models and common dependencies like lodash.

Promises make it a bit tricky. Maybe https://github.com/ef4/async-repl

ChartBaker should ensure git repository exists at repoDir

Otherwise the commands it runs might commit to the code repository, which is no good

Basic version history for chart configuration

Talking to Esteban, there a couple of cases where published charts have been broken in some way by a mistake in editing and it was tricky to find out what the original settings were. It'd be great to have some way of seeing previous published versions of a chart configuration and rolling back.

This doesn't have to be fancy, just fulfil the function of being able to undo. Storing old jsons in a table somewhere with a date on them and a little interface in the admin editor to select them is all that is needed.

Configure for indexing of entries in Google Scholar

OWID entries are approximately analogous to journal articles. It would make sense to have them indexed in Google Scholar!

This means adding the appropriate meta tags: https://scholar.google.com/intl/en/scholar/inclusion.html#indexing

We can use distill.pub as an example https://distill.pub/2018/differentiable-parameterizations/

It might make sense to look at this in conjunction with getting DOIs.

Smarter default entity selection

Max writes:

– It'd be good if only countries would be selected and not world regions in any definition (Only entities that have an OWID country name should be displayed but not 'middle-income countries' or 'highly indebted low income countries' or similar specific categories.) One exception to this would be that 'World' should be displayed when chosen at random. And the second exception would be that world regions should be shown if the dataset does not contain data on countries and only on world regions; More generally if no country data is available in a dataset than the displayed entities are completely random, as right now.)

– It might be nice to show 5 countries by default. One (at random) from each world region: Europe, Africa, Asia, Oceania, Americas.

– We could think about whether it makes sense to only show countries with a population of over 1 or 5 million.

---

In other words, when creating a new chart:

• Select 5 default keys to show, prioritizing World if available, then high-population countries from different world regions (pregenerate/hardcode a list of candidates), then non-country entities

Fix Slack error reporting

The admin server is supposed to hit a webhook when it encounters an unhandled error to post it to Slack, using an express middleware, but it doesn't seem to be doing so. Probably the other error handling in app.tsx that sends stack traces down to the client is interfering with it, it would be good if they played nice together.

Fix the borders of Somalia and make tiny countries findable on maps

Kinda two issues, but they both involve changing the underlying map topography so rolling them together.

Max provides this static image as an example of what maps might look like:

• Somalia

Our maps (like many) are based on the topojson from https://github.com/topojson/world-atlas, which in turn comes from the Natural Earth vector data here: http://www.naturalearthdata.com/downloads/

The borders in this topojson distinguish Somalia from Somaliland: https://en.wikipedia.org/wiki/Somaliland This self-declared state is not widely recognized and readers are sometimes unhappy that we implicitly recognize them in the border configuration-- and we don't tend to have data for Somaliland anyway. That whole bit should just be Somalia like how Google Maps shows it.

• Tiny countries

There are many small nations for which we have data but which occupy a very small geographic area, so that they are hard to see or hover over on the interactive map (for example, Singapore). These should be represented by small circles, ensuring that all countries occupy a certain minimum visible area on the map.

Implementation

Ideally we would go and edit the original geojson to make these changes and then rebuild the topojson. From a brief look around I couldn't quite figure out how to do this, but it should be possible.

Better export script for allowing new devs to test charts

Port testing pages to node

Showcase data quarterly

Is it possible to import and showcase data as quarterly? At the moment we can only import as yearly,

Full-node update

Investigate further: possibility of using SVG file directly as the base for choropleth maps

Is there anything stopping us from doing that in practice? How does the performance/filesize compare with topojson? A little weird with React, but it might work.

Tagging of individual charts

A high-priority feature is creating a public index of all of our charts. For this, we need to know the categories that individual charts fall under.

I was originally thinking we would categorize charts based on either the page they are embedded in or the dataset they originate from. But after talking with people it becomes clear that we do want charts to have their own categories, and maybe just use one of those reference points to assign an initial set to all the existing charts.

The database structure for this is straightforward (same as for datasets). Then it is just a matter of doing the UI for it.

Support for data time ranges other than annual

This is a tricky one since it requires carefully rethinking all aspects of the grapher (the importer and database as well as rendering). We currently get a lot of systemic simplicity out of being able to assume that all time data is in the form of integer years.

However, we will need to implement this at some point to fulfill our core goals of accurately collating all global development data-- there are some datasets we cannot represent in our database right now.