GithubHelp home page GithubHelp logo

owid / owid-grapher Goto Github PK

View Code? Open in Web Editor NEW
1.3K 1.3K 227.0 130.67 MB

A platform for creating interactive data visualizations

Home Page: https://ourworldindata.org

License: MIT License

JavaScript 1.02% Shell 0.51% TypeScript 91.74% HTML 0.19% SCSS 6.10% Dockerfile 0.01% Makefile 0.23% Python 0.18% Handlebars 0.02%
data-visualization react typescript

owid-grapher's People

Contributors

aaldaber avatar bhch7051 avatar bnjmacdonald avatar breck7 avatar danielgavrilov avatar danyx23 avatar dependabot-preview[bot] avatar dependabot[bot] avatar edomt avatar hannahritchie avatar ikesau avatar jasoncrawford avatar larsyencken avatar lucasrodes avatar mahmoudhamdy02 avatar marcelgerber avatar marigold avatar markledwich2 avatar mathisonian avatar mlbrgl avatar owidbot avatar pabloarosado avatar rakyi avatar samizdatco avatar shaahmad avatar shafy avatar sherin avatar sophiamersmann avatar toni-sharpe avatar zdenekhynek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

owid-grapher's Issues

Basic version history for chart configuration

Talking to Esteban, there a couple of cases where published charts have been broken in some way by a mistake in editing and it was tricky to find out what the original settings were. It'd be great to have some way of seeing previous published versions of a chart configuration and rolling back.

This doesn't have to be fancy, just fulfil the function of being able to undo. Storing old jsons in a table somewhere with a date on them and a little interface in the admin editor to select them is all that is needed.

Indexing of static visualizations in the grapher

The grapher has the nice property that the code can keep close track of all the interactive visualizations, make them searchable and show the data they are sourced from, etc. It would be neat if we could also keep track of visualizations produced outside the grapher in the same way, at least to some extent.

The most minimal version would simply be allowing images to be uploaded in the admin with a title, which are published to an image host somewhere. We could then make the images searchable by that title.

Error importing new data

The images below is a screenshot of the errors I got when I tried to import new data into the application. Could you help explain what the problem might be. Thanks

image

image

Slope charts should have region legends like scatterplots

Fix Slack error reporting

The admin server is supposed to hit a webhook when it encounters an unhandled error to post it to Slack, using an express middleware, but it doesn't seem to be doing so. Probably the other error handling in app.tsx that sends stack traces down to the client is interfering with it, it would be good if they played nice together.

Public charts index page

We have a lot of visualizations now; let's make it easier for people to find the ones they are looking for.

The minimal version of this would be a page at https://ourworldindata.org/charts or similar which lists the titles of all grapher visualizations and links through to them, with a client-side search function similar to what we use on the admin chart index.

A later version might also include static visualizations or make use of image previews.

Technical considerations:

  • It's a page in the Wordpress domain but needs to access the grapher db directly
    • So we want to merge the owid-grapher and owid-theme builds for this
  • The page needs to be updated whenever a chart is published or unpublished
  • Client-side search is acceptable for now (just a few thousand strings) but if we incorporate more information later we will probably then need an API service of some kind

Generate a world map on the country standardizer tool

From Max:

Not a necessary feature at all but a little idea which makes the tool look nice and which is relatively easy to set up with what we have built already for Our World in Data: It should be possible that when someone uploads a dataset into the country standardizer that the tool produces a world map with that data. That way people would already have a way to explore the data and they can save world map charts of it.


Feature to add in Country Tool: http://localhost:3030/admin/standardize
This is similar to the preview shown in the Variables page: http://localhost:3030/admin/variables

Showcase data quarterly

Is it possible to import and showcase data as quarterly? At the moment we can only import as yearly,

Use the stacked area hover behavior for other charts as well

Stacked area hover tooltips look like this:

screen shot 2018-09-25 at 5 27 18 pm

Hovering over the chart creates a vertical line at a given x value and then shows the y values for each series in descending order.

I quite like this because it lets you easily directly compare all the different values for a year. It occurs to me that there's no reason this behavior couldn't apply just the same to Line and Stacked Bar charts as well-- the only difference would be that it wouldn't display a total in the former case.

Slope charts should allow highlighting and searching entities like scatterplots

Single-year stacked bar charts

From Hannah:

Max says that Wendy from Core-Econ writes:

"Stacked Bar Charts
When creating bar graphs, the bars are coded to be horizontal. (e.g. see https://ourworldindata.org/grapher/wages-relative-to-the-price-of-energy) Nonetheless, it would be great to have the option of vertical bars. – Non-priority

There is no option for stacked bar graph. Therefore, we cannot input e.g. the following graph: https://core-econ.org/the-economy/book/text/05.html#figure-5-16
Non Priority"

This is something that would be great for us to have also. Sherin amazingly made us stacked bar charts by year (with year on the x-axis) (e.g. https://ourworldindata.org/grapher/deaths-from-natural-disasters-by-type). These are incredibly useful.

What would also be useful is to have stacked bar charts where time is not on the y-axis. In other words, single-year views with countries or other entries as the bars. For example, you could have a country for each stacked bar (you can add and remove countries as we can with discrete bar charts already) with the stacked bar showing meat consumption by type, or energy production by source, or causes of death. This lets you directly compare a number of countries in a given year.

E.g (but maybe best to turn it 90 degrees so the countries are on the x-axis?):

preview-full-screen shot 2018-11-22 at 06 39 17

The other possibility Esteban previously raised as helpful here is whether it's easy to have multiple bars (side-by-side) for a given country in a given year. For example, for each country having male and female bars next to each other (with stacks of how they spend their time, or stacks of what they die from). I don't know if this one is as easily doable, but one possibility for the future. Esteban would be happy!

E.g.
preview-full-screen shot 2018-11-22 at 06 37 55

Thanks guys!

no import page

With c55431d, I see an empty panel when clicking on "Import New Data". The javascript console says TypeError: props.sourceTemplate is null at localhost:8000/import, line 105, which says new Apps.View.ImportView(importerData).

Node scripts with calls to shell commands should fail fast

Currently we use shell.exec from shelljs: https://github.com/shelljs/shelljs

However, shell.exec doesn't throw an exception on a non-zero exit code, and so things can fail in weird ways. We should replace these execs with stricter ones and (importantly) make sure there's no code that implicitly relies on the existing behavior to work.

Is there a better node shell library, maybe one that uses Promises so we can use async/await too?

Allow multiple regression lines for scatterplots

This is a nice straightforward feature request from the core-econ team. Some editor UI, small changes to code and possibly a migration to restructure existing chart configuration slightly should be all that is needed.

Tagging of individual charts

A high-priority feature is creating a public index of all of our charts. For this, we need to know the categories that individual charts fall under.

I was originally thinking we would categorize charts based on either the page they are embedded in or the dataset they originate from. But after talking with people it becomes clear that we do want charts to have their own categories, and maybe just use one of those reference points to assign an initial set to all the existing charts.

The database structure for this is straightforward (same as for datasets). Then it is just a matter of doing the UI for it.

Don't allow selecting countries without renderable data on scatterplots or slope charts

This is a bug that has been around for a while, but you can see it on e.g. https://ourworldindata.org/grapher/number-of-child-deaths-slope-chart Where it's possible to select Afghanistan but it doesn't appear on the slope chart.

This is because the data selector looks directly at the data to determine what entities are available without reference to the chart type, so it doesn't know whether it's renderable or not. Determining this list of entities should be the responsibility of the chart transform, which has that information.

Not sure what the best way to get that info from the transform to the selector UI is... maybe expose it via an interface on activeTransform?

Tracking embed locations for each chart

We have the ability to set an origin url for each chart in the grapher so that the chart can link back to the text. However, this requires manual work and people don't always remember to do it. It would be neat if the code could automatically know which of our pages a chart is embedded.

I think the way this would work is that each time a post is saved, the code goes through the html and picks out any embeds, then updates a database table indicating the relationship. If it is the first such relationship, and the chart has no canonical origin url, it also updates the chart setting the origin url to that page.

This one is a little tricky because of the degree of integration between articles and charts it requires. Probably something to do in conjunction with replacing the Wordpress editor.

Split the frontend code from owid-grapher into a standalone visualization library

Often people ask what we use to make interactive charts. I can point them at this repo but they can't really just go and adapt it for their own purposes, since everything is tied together as a single web app.

I would like to extract the actual data visualization code from this repository and make it a separate npm dependency that people can easily reuse. Something like nvd3, which we used to depend on ourselves until we found it too limited.

I think the grapher code has some interesting properties that could make it a useful addition to this space, particularly in comparison to older stuff like nvd3:

  • Fully responsive, including SVG text wrapping and label overlap detection
  • Local SVG and PNG exports with text annotation
  • Universal rendering: rare datavis library that can run entirely under node
  • Nice Mobx interface for efficient state mutation (chart.title = "foo")
  • Designed to integrate into other automated systems (like the chart editor)
  • We have the funding and incentive to maintain it for 3+ years

I think we could present it as being a "zero configuration" kind of interface, since that's what it's sort of already designed to be: you give it some data and it infers an interactive chart from it. While some customization options would of course be available on the chart itself, the ideal is that you customize it by annotating the data to give the grapher more information to work with.

We don't want to pursue the kind of flexibility of a system like Vega. The grapher will always do a limited set of things, but we want it to be the best option for doing the things that it can do.

To allow for easy prototyping, it should come in both the developer-focused, npm variant, and also a CDN script tag bundle you can easily include and play with. Also it should have a cool name!

Dual-point interactive timeline on maps

Something we've wanted to do for a while is add interactive timelines to each of the chart types which allow you to specify both a start and end year, then show you the chart for that period.

We currently have this only for scatterplots: https://ourworldindata.org/grapher/life-expectancy-vs-health-expenditure

For choropleth maps, in multi-year mode it would color the map according to the change in values between the two selected years.

Some thought will need to be put into how the legend and color selection works for this!

I've never been suuper happy with the usability of the dual-point interface on scatterplots (particularly switching between single and multi year modes). If we can think of improvements to it that would also be good.

Improvements to the label placement algorithm for line charts

A while ago I implemented a positional labelling system based on some of the charts in the book "The Functional Art". It looks like this:

screen shot 2018-07-30 at 1 33 06 pm

This HeightedLegend algorithm works by placing all the labels at the height of the line, finding all overlaps, and then shifting all overlaps either down or up depending on which direction has available space. In most cases this works well and finds an acceptable solution.

However, there are occasional edge cases like this where it cannot find a solution: https://ourworldindata.org/grapher/mispy-overlap-test

global-incidence-and-prevalence-of-tuberculosis

This is because the algorithm always moves all overlapping labels either down or up, it cannot do both simultaneously because it would then need to ensure that it did not create more overlaps in the process of repositioning.

Label placement is a complex problem in computer science, but since this particular problem is 1-dimensional (only the height varies) the algorithm does not need to be super optimized and so I think it should be possible to find a better solution that resolves this edge case (and also has improved labelling behavior in the case when there are clusters in the center).

Fix the borders of Somalia and make tiny countries findable on maps

Kinda two issues, but they both involve changing the underlying map topography so rolling them together.

Max provides this static image as an example of what maps might look like:

preview-lightbox-screen shot 2018-05-30 at 15 54 36 3

  • Somalia

Our maps (like many) are based on the topojson from https://github.com/topojson/world-atlas, which in turn comes from the Natural Earth vector data here: http://www.naturalearthdata.com/downloads/

The borders in this topojson distinguish Somalia from Somaliland: https://en.wikipedia.org/wiki/Somaliland This self-declared state is not widely recognized and readers are sometimes unhappy that we implicitly recognize them in the border configuration-- and we don't tend to have data for Somaliland anyway. That whole bit should just be Somalia like how Google Maps shows it.

screen shot 2018-07-30 at 7 41 07 pm

  • Tiny countries

There are many small nations for which we have data but which occupy a very small geographic area, so that they are hard to see or hover over on the interactive map (for example, Singapore). These should be represented by small circles, ensuring that all countries occupy a certain minimum visible area on the map.

screen shot 2018-07-30 at 7 42 36 pm

Implementation

Ideally we would go and edit the original geojson to make these changes and then rebuild the topojson. From a brief look around I couldn't quite figure out how to do this, but it should be possible.

Support for data time ranges other than annual

This is a tricky one since it requires carefully rethinking all aspects of the grapher (the importer and database as well as rendering). We currently get a lot of systemic simplicity out of being able to assume that all time data is in the form of integer years.

However, we will need to implement this at some point to fulfill our core goals of accurately collating all global development data-- there are some datasets we cannot represent in our database right now.

New chart type: stacked bar chart

Hannah writes:

We have just been discussing which new entries (and updates) which would be good to try to get some material out on in the near future. There are quite a few for which stacked bar charts would be really helpful.

For example, these natural catastrophe ones: https://ourworldindata.org/natural-catastrophes/#decadal-deaths-and-death-rates

For a new causes of death entry, something like this with deaths by age:

image
image
image


To implement this, start by copying the Discrete Bar chart type and go from there:

https://github.com/owid/owid-grapher/blob/master/js/charts/DiscreteBarTransform.ts
https://github.com/owid/owid-grapher/blob/master/js/charts/DiscreteBarChart.tsx

These are the relevant hooks for adding new chart types:

https://github.com/owid/owid-grapher/blob/master/js/charts/ChartConfig.tsx#L321
https://github.com/owid/owid-grapher/blob/master/js/charts/ChartTab.tsx#L34
https://github.com/owid/owid-grapher/blob/master/js/admin/EditorBasicTab.tsx#L103

Logo display

When uploading a logo, where should it display? It is not displaying on the graph.

Missing files in the zip archive

I downloaded the zip archive. But by using it, I notice that some files are missing.
Example of the file /vendor/autoload.php

Error on import new data page

I get this error when I open the page (import new data):

local.ERROR: ErrorException: Trying to get property of non-object in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0ca63521.php:137

and

Next ErrorException: Trying to get property of non-object (View: /var/www/owid-grapher/resources/views/import/index.blade.php) in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0ca63521.php:137

Looking a little bit I think I got it fixed using the August 5 schema version "Add dataset (sub) categories to schema dump" and updating the "logos" table with the one you have in the last update (Dec 23, 2016)

--
-- Table structure for table `logos`
--

DROP TABLE IF EXISTS `logos`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `logos` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(100) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `svg` text NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

Now apparently it works and loads everything, but I have not had time to prove it yet. I hope it helps

Smarter default entity selection

Max writes:

– It'd be good if only countries would be selected and not world regions in any definition (Only entities that have an OWID country name should be displayed but not 'middle-income countries' or 'highly indebted low income countries' or similar specific categories.) One exception to this would be that 'World' should be displayed when chosen at random. And the second exception would be that world regions should be shown if the dataset does not contain data on countries and only on world regions; More generally if no country data is available in a dataset than the displayed entities are completely random, as right now.)

– It might be nice to show 5 countries by default. One (at random) from each world region: Europe, Africa, Asia, Oceania, Americas.

– We could think about whether it makes sense to only show countries with a population of over 1 or 5 million.


In other words, when creating a new chart:

  • Select 5 default keys to show, prioritizing World if available, then high-population countries from different world regions (pregenerate/hardcode a list of candidates), then non-country entities

Remove legacy metadata columns from variables table

We chatted with @mispy about this—some columns in the variables table were originally used to store metadata, but due to many different metadata formats, originalMetadata is now used to store all metadata as JSON.

Some of these columns are: unit, shortUnit, code, coverage, timespan. Ideally, if they are useful and cannot be derived from anything else, they should be moved to originalMetadata as JSON fields.

Error when editing dataset variable.

When editing and changing a dataset variable we get the following error:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'updated_at' in 'field list' (SQL: updatedata_valuessetvalue= 54.84,updated_at= 2016-08-19 06:20:22 whereid= 3413091)

Port the country name standardizer to JavaScript

Aibek wrote this useful tool for standardizing country names between different formats, a common problem in working with economics datasets: https://github.com/owid/owid-grapher/tree/master/country_name_tool

This is currently server-side code, but we'd like to do most of it client-side. It should be possible to preload the necessary entity data and process the CSV entirely in the browser.

I think the only server-side step should be the remembering of choices in the case when manual resolution is needed.

Once ported to client-side JS, we could then think about moving the standardizer out of the admin and making it a standalone public tool.

Error on dataset import page

Hi,
I get this error when going to the "import new data" menu:

Next exception 'ErrorException' with message 'Trying to get property of non-object (View: /var/www/owid-grapher/resources/views/import/index.blade.php)' in /var/www/owid-grapher/storage/framework/views/11161acccbf6d47c6b03d0c410cd104b0c104b0ca63521.php:132

I also notice that there is no categories, and no way to add categories.

I followed the installation instructions in the README.md file. Am i missing something?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.