owid / owid-grapher Goto Github PK

The slope chart transform code currently uses this Observations thing I was experimenting with. I ended up deciding not to use that and process data in a simpler way; it should work directly from DimensionWithData.

Talking to Esteban, there a couple of cases where published charts have been broken in some way by a mistake in editing and it was tricky to find out what the original settings were. It'd be great to have some way of seeing previous published versions of a chart configuration and rolling back.

This doesn't have to be fancy, just fulfil the function of being able to undo. Storing old jsons in a table somewhere with a date on them and a little interface in the admin editor to select them is all that is needed.

The grapher has the nice property that the code can keep close track of all the interactive visualizations, make them searchable and show the data they are sourced from, etc. It would be neat if we could also keep track of visualizations produced outside the grapher in the same way, at least to some extent.

The most minimal version would simply be allowing images to be uploaded in the admin with a title, which are published to an image host somewhere. We could then make the images searchable by that title.

At the moment it's always maps which doesn't work for non-geographic variables. Should pick like:

Mappable entities => map
Single-year non-mappable data => discrete bar chart
Multi-year non-mappable data => line chart

So contributors can always have the most recent schema+data

The images below is a screenshot of the errors I got when I tried to import new data into the application. Could you help explain what the problem might be. Thanks

This one should be reasonably straightforward.

We want to take the little legend from the scatterplot chart type, e.g. https://ourworldindata.org/grapher/urbanization-vs-gdp

And put it on the "slope chart" chart type, e.g. https://ourworldindata.org/grapher/access-to-a-financial-account-or-services

The relevant files to look at:

https://github.com/owid/owid-grapher/blob/master/js/charts/ScatterPlot.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/ScatterColorLegend.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/SlopeChart.tsx

The admin server is supposed to hit a webhook when it encounters an unhandled error to post it to Slack, using an express middleware, but it doesn't seem to be doing so. Probably the other error handling in app.tsx that sends stack traces down to the client is interfering with it, it would be good if they played nice together.

From Max:

If you download the data behind this chart https://ourworldindata.org/grapher/gini-index-around-2015-1990-2015-countries-vs-gini-index-around-1990-1990-2015-countries

it gives you the population figures for all countries in all years for the last two centuries.

Let’s make a note to change this!

Ideally, the extra data shouldn't even be served in the JSON chart data since it can't be used in any way?

We have a lot of visualizations now; let's make it easier for people to find the ones they are looking for.

The minimal version of this would be a page at https://ourworldindata.org/charts or similar which lists the titles of all grapher visualizations and links through to them, with a client-side search function similar to what we use on the admin chart index.

A later version might also include static visualizations or make use of image previews.

Technical considerations:

• It's a page in the Wordpress domain but needs to access the grapher db directly
  • So we want to merge the owid-grapher and owid-theme builds for this
• The page needs to be updated whenever a chart is published or unpublished
• Client-side search is acceptable for now (just a few thousand strings) but if we incorporate more information later we will probably then need an API service of some kind

From Max:

Not a necessary feature at all but a little idea which makes the tool look nice and which is relatively easy to set up with what we have built already for Our World in Data: It should be possible that when someone uploads a dataset into the country standardizer that the tool produces a world map with that data. That way people would already have a way to explore the data and they can save world map charts of it.

Feature to add in Country Tool: http://localhost:3030/admin/standardize
This is similar to the preview shown in the Variables page: http://localhost:3030/admin/variables

Is it possible to import and showcase data as quarterly? At the moment we can only import as yearly,

e.g. https://ourworldindata.org/grapher/mountain-green-cover-index is currently weird

Otherwise the commands it runs might commit to the code repository, which is no good

Google recently announced that it will show tabular data in search results if you provide the metadata for it: https://www.blog.google/outreach-initiatives/google-news-initiative/making-it-easier-discover-data-search/

This would be particularly useful for when we make public dataset pages later, but I think we could potentially also go ahead and add it to the static html build for grapher charts now.

Stacked area hover tooltips look like this:

Hovering over the chart creates a vertical line at a given x value and then shows the y values for each series in descending order.

I quite like this because it lets you easily directly compare all the different values for a year. It occurs to me that there's no reason this behavior couldn't apply just the same to Line and Stacked Bar charts as well-- the only difference would be that it wouldn't display a total in the former case.

We want to take the little legend from the scatterplot chart type, e.g. https://ourworldindata.org/grapher/urbanization-vs-gdp

And put it on the "slope chart" chart type, e.g. https://ourworldindata.org/grapher/access-to-a-financial-account-or-services

Performance is an important consideration here; we want to try to keep the number of SVG elements and updates to them to a minimum.

The relevant files to look at:

https://github.com/owid/owid-grapher/blob/master/js/charts/ScatterPlot.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/ScatterColorLegend.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/SlopeChart.tsx
https://github.com/owid/owid-grapher/blob/master/js/charts/ControlsFooter.tsx

Is there anything stopping us from doing that in practice? How does the performance/filesize compare with topojson? A little weird with React, but it might work.

From Hannah:

Max says that Wendy from Core-Econ writes:

"Stacked Bar Charts
When creating bar graphs, the bars are coded to be horizontal. (e.g. see https://ourworldindata.org/grapher/wages-relative-to-the-price-of-energy) Nonetheless, it would be great to have the option of vertical bars. – Non-priority

There is no option for stacked bar graph. Therefore, we cannot input e.g. the following graph: https://core-econ.org/the-economy/book/text/05.html#figure-5-16
Non Priority"

This is something that would be great for us to have also. Sherin amazingly made us stacked bar charts by year (with year on the x-axis) (e.g. https://ourworldindata.org/grapher/deaths-from-natural-disasters-by-type). These are incredibly useful.

What would also be useful is to have stacked bar charts where time is not on the y-axis. In other words, single-year views with countries or other entries as the bars. For example, you could have a country for each stacked bar (you can add and remove countries as we can with discrete bar charts already) with the stacked bar showing meat consumption by type, or energy production by source, or causes of death. This lets you directly compare a number of countries in a given year.

E.g (but maybe best to turn it 90 degrees so the countries are on the x-axis?):

The other possibility Esteban previously raised as helpful here is whether it's easy to have multiple bars (side-by-side) for a given country in a given year. For example, for each country having male and female bars next to each other (with stacks of how they spend their time, or stacks of what they die from). I don't know if this one is as easily doable, but one possibility for the future. Esteban would be happy!

E.g.

Thanks guys!

OWID entries are approximately analogous to journal articles. It would make sense to have them indexed in Google Scholar!

This means adding the appropriate meta tags: https://scholar.google.com/intl/en/scholar/inclusion.html#indexing

We can use distill.pub as an example https://distill.pub/2018/differentiable-parameterizations/

It might make sense to look at this in conjunction with getting DOIs.

With c55431d, I see an empty panel when clicking on "Import New Data". The javascript console says TypeError: props.sourceTemplate is null at localhost:8000/import, line 105, which says new Apps.View.ImportView(importerData).

Would look nicer I think. Maybe? I'll think about it.

Currently we use shell.exec from shelljs: https://github.com/shelljs/shelljs

However, shell.exec doesn't throw an exception on a non-zero exit code, and so things can fail in weird ways. We should replace these execs with stricter ones and (importantly) make sure there's no code that implicitly relies on the existing behavior to work.

Is there a better node shell library, maybe one that uses Promises so we can use async/await too?

This is a nice straightforward feature request from the core-econ team. Some editor UI, small changes to code and possibly a migration to restructure existing chart configuration slightly should be all that is needed.

A high-priority feature is creating a public index of all of our charts. For this, we need to know the categories that individual charts fall under.

I was originally thinking we would categorize charts based on either the page they are embedded in or the dataset they originate from. But after talking with people it becomes clear that we do want charts to have their own categories, and maybe just use one of those reference points to assign an initial set to all the existing charts.

The database structure for this is straightforward (same as for datasets). Then it is just a matter of doing the UI for it.

This is a bug that has been around for a while, but you can see it on e.g. https://ourworldindata.org/grapher/number-of-child-deaths-slope-chart Where it's possible to select Afghanistan but it doesn't appear on the slope chart.

This is because the data selector looks directly at the data to determine what entities are available without reference to the chart type, so it doesn't know whether it's renderable or not. Determining this list of entities should be the responsibility of the chart transform, which has that information.

Not sure what the best way to get that info from the transform to the selector UI is... maybe expose it via an interface on activeTransform?

We have the ability to set an origin url for each chart in the grapher so that the chart can link back to the text. However, this requires manual work and people don't always remember to do it. It would be neat if the code could automatically know which of our pages a chart is embedded.

I think the way this would work is that each time a post is saved, the code goes through the html and picks out any embeds, then updates a database table indicating the relationship. If it is the first such relationship, and the chart has no canonical origin url, it also updates the chart setting the origin url to that page.

This one is a little tricky because of the degree of integration between articles and charts it requires. Probably something to do in conjunction with replacing the Wordpress editor.

Often people ask what we use to make interactive charts. I can point them at this repo but they can't really just go and adapt it for their own purposes, since everything is tied together as a single web app.

I would like to extract the actual data visualization code from this repository and make it a separate npm dependency that people can easily reuse. Something like nvd3, which we used to depend on ourselves until we found it too limited.

I think the grapher code has some interesting properties that could make it a useful addition to this space, particularly in comparison to older stuff like nvd3:

• Fully responsive, including SVG text wrapping and label overlap detection
• Local SVG and PNG exports with text annotation
• Universal rendering: rare datavis library that can run entirely under node
• Nice Mobx interface for efficient state mutation (chart.title = "foo")
• Designed to integrate into other automated systems (like the chart editor)
• We have the funding and incentive to maintain it for 3+ years

I think we could present it as being a "zero configuration" kind of interface, since that's what it's sort of already designed to be: you give it some data and it infers an interactive chart from it. While some customization options would of course be available on the chart itself, the ideal is that you customize it by annotating the data to give the grapher more information to work with.

We don't want to pursue the kind of flexibility of a system like Vega. The grapher will always do a limited set of things, but we want it to be the best option for doing the things that it can do.

To allow for easy prototyping, it should come in both the developer-focused, npm variant, and also a CDN script tag bundle you can easily include and play with. Also it should have a cool name!

Something we've wanted to do for a while is add interactive timelines to each of the chart types which allow you to specify both a start and end year, then show you the chart for that period.

We currently have this only for scatterplots: https://ourworldindata.org/grapher/life-expectancy-vs-health-expenditure

For choropleth maps, in multi-year mode it would color the map according to the change in values between the two selected years.

Some thought will need to be put into how the legend and color selection works for this!

I've never been suuper happy with the usability of the dual-point interface on scatterplots (particularly switching between single and multi year modes). If we can think of improvements to it that would also be good.

A while ago I implemented a positional labelling system based on some of the charts in the book "The Functional Art". It looks like this:

This HeightedLegend algorithm works by placing all the labels at the height of the line, finding all overlaps, and then shifting all overlaps either down or up depending on which direction has available space. In most cases this works well and finds an acceptable solution.

However, there are occasional edge cases like this where it cannot find a solution: https://ourworldindata.org/grapher/mispy-overlap-test

This is because the algorithm always moves all overlapping labels either down or up, it cannot do both simultaneously because it would then need to ensure that it did not create more overlaps in the process of repositioning.

Label placement is a complex problem in computer science, but since this particular problem is 1-dimensional (only the height varies) the algorithm does not need to be super optimized and so I think it should be possible to find a better solution that resolves this edge case (and also has improved labelling behavior in the case when there are clusters in the center).

Kinda two issues, but they both involve changing the underlying map topography so rolling them together.

Max provides this static image as an example of what maps might look like:

• Somalia

Our maps (like many) are based on the topojson from https://github.com/topojson/world-atlas, which in turn comes from the Natural Earth vector data here: http://www.naturalearthdata.com/downloads/

The borders in this topojson distinguish Somalia from Somaliland: https://en.wikipedia.org/wiki/Somaliland This self-declared state is not widely recognized and readers are sometimes unhappy that we implicitly recognize them in the border configuration-- and we don't tend to have data for Somaliland anyway. That whole bit should just be Somalia like how Google Maps shows it.

• Tiny countries

There are many small nations for which we have data but which occupy a very small geographic area, so that they are hard to see or hover over on the interactive map (for example, Singapore). These should be represented by small circles, ensuring that all countries occupy a certain minimum visible area on the map.

Implementation

Ideally we would go and edit the original geojson to make these changes and then rebuild the topojson. From a brief look around I couldn't quite figure out how to do this, but it should be possible.

This is a tricky one since it requires carefully rethinking all aspects of the grapher (the importer and database as well as rendering). We currently get a lot of systemic simplicity out of being able to assume that all time data is in the form of integer years.

However, we will need to implement this at some point to fulfill our core goals of accurately collating all global development data-- there are some datasets we cannot represent in our database right now.

Would be neat to be able to validate API calls at compile time. Using something like https://github.com/rawrmaan/restyped perhaps

For core-econ people

Hannah writes:

We have just been discussing which new entries (and updates) which would be good to try to get some material out on in the near future. There are quite a few for which stacked bar charts would be really helpful.

For example, these natural catastrophe ones: https://ourworldindata.org/natural-catastrophes/#decadal-deaths-and-death-rates

For a new causes of death entry, something like this with deaths by age:

To implement this, start by copying the Discrete Bar chart type and go from there:

https://github.com/owid/owid-grapher/blob/master/js/charts/DiscreteBarTransform.ts
https://github.com/owid/owid-grapher/blob/master/js/charts/DiscreteBarChart.tsx

These are the relevant hooks for adding new chart types:

https://github.com/owid/owid-grapher/blob/master/js/charts/ChartConfig.tsx#L321
https://github.com/owid/owid-grapher/blob/master/js/charts/ChartTab.tsx#L34
https://github.com/owid/owid-grapher/blob/master/js/admin/EditorBasicTab.tsx#L103

Like these maps: https://flourish.studio/2018/04/25/london-ward-maps/

We can know exactly where the value should be positionally, so we might as well be precise

When uploading a logo, where should it display? It is not displaying on the graph.

Always find myself missing rails console from my Ruby days. Can we get something similar with node? Just needs to open a basic REPL with the global scope already populated with most of the project models and common dependencies like lodash.

Promises make it a bit tricky. Maybe https://github.com/ef4/async-repl

I downloaded the zip archive. But by using it, I notice that some files are missing.
Example of the file /vendor/autoload.php

Like with scatterplots: https://ourworldindata.org/grapher/life-expectancy-vs-health-expenditure

For single-year mode, line charts could become bar charts.

I think this one should be reasonably straightforward. Backwards compatibility may have some interesting edge cases.

I get this error when I open the page (import new data):

local.ERROR: ErrorException: Trying to get property of non-object in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0ca63521.php:137

and

Next ErrorException: Trying to get property of non-object (View: /var/www/owid-grapher/resources/views/import/index.blade.php) in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0ca63521.php:137

Looking a little bit I think I got it fixed using the August 5 schema version "Add dataset (sub) categories to schema dump" and updating the "logos" table with the one you have in the last update (Dec 23, 2016)

--
-- Table structure for table `logos`
--

DROP TABLE IF EXISTS `logos`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `logos` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(100) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `svg` text NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

Now apparently it works and loads everything, but I have not had time to prove it yet. I hope it helps

Max writes:

– It'd be good if only countries would be selected and not world regions in any definition (Only entities that have an OWID country name should be displayed but not 'middle-income countries' or 'highly indebted low income countries' or similar specific categories.) One exception to this would be that 'World' should be displayed when chosen at random. And the second exception would be that world regions should be shown if the dataset does not contain data on countries and only on world regions; More generally if no country data is available in a dataset than the displayed entities are completely random, as right now.)

– It might be nice to show 5 countries by default. One (at random) from each world region: Europe, Africa, Asia, Oceania, Americas.

– We could think about whether it makes sense to only show countries with a population of over 1 or 5 million.

In other words, when creating a new chart:

• Select 5 default keys to show, prioritizing World if available, then high-population countries from different world regions (pregenerate/hardcode a list of candidates), then non-country entities

We chatted with @mispy about this—some columns in the variables table were originally used to store metadata, but due to many different metadata formats, originalMetadata is now used to store all metadata as JSON.

Some of these columns are: unit, shortUnit, code, coverage, timespan. Ideally, if they are useful and cannot be derived from anything else, they should be moved to originalMetadata as JSON fields.

When editing and changing a dataset variable we get the following error:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'updated_at' in 'field list' (SQL: updatedata_valuessetvalue= 54.84,updated_at= 2016-08-19 06:20:22 whereid= 3413091)

Aibek wrote this useful tool for standardizing country names between different formats, a common problem in working with economics datasets: https://github.com/owid/owid-grapher/tree/master/country_name_tool

This is currently server-side code, but we'd like to do most of it client-side. It should be possible to preload the necessary entity data and process the CSV entirely in the browser.

I think the only server-side step should be the remembering of choices in the case when manual resolution is needed.

Once ported to client-side JS, we could then think about moving the standardizer out of the admin and making it a standalone public tool.

Hi,
I get this error when going to the "import new data" menu:

Next exception 'ErrorException' with message 'Trying to get property of non-object (View: /var/www/owid-grapher/resources/views/import/index.blade.php)' in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0c104b0ca63521.php:132

I also notice that there is no categories, and no way to add categories.

I followed the installation instructions in the README.md file. Am i missing something?