Moved from openspending/openspending#667

Next to making visualizations embedable - offer a download button to download an image of the current view...

In 37ff613 I've extracted the column generation algorithm to a public method, so I could add the "Yearly Change" column in openspending/slovakia.openspending.org@9a545ba#L1R15.

The problem is that, when creating an AggregateTable, we return a jQuery's Deferred that'll be resolved when yepnope finishes loading the widget's requirements (https://github.com/openspending/openspendingjs/blob/bb767920270ba3f87a1544451a4e49813f611158/widgets/aggregate_table/main.js#L197-L215). After that, we call the init() method.

This method then does some initializations, and load the dataset's model.json. When it finishes, it calls the update() method (https://github.com/openspending/openspendingjs/blob/bb767920270ba3f87a1544451a4e49813f611158/widgets/aggregate_table/main.js#L191). It then goes on and sets up and initializes dataTable.

The problem is that to call init(), we're using the Deferred (https://github.com/openspending/openspendingjs/blob/bb767920270ba3f87a1544451a4e49813f611158/widgets/aggregate_table/main.js#L201). The same as the user of the AggregateTable. So our deferred callback runs before theirs. If they change the columns in their callback (as in https://github.com/openspending/slovakia.openspending.org/blob/9a545ba83fba1eac57724ef2138dd9a43f003e98/js/treetable.js#L15), it might happen after we called update() and created the dataTable, so the column isn't added.

The workaround I did in Slovakia was to call update() again, in https://github.com/openspending/slovakia.openspending.org/blob/9a545ba83fba1eac57724ef2138dd9a43f003e98/js/treetable.js#L22-L25. I added the check to be sure the dataTable is loaded before we try to add the columns (if it wasn't, everything will work OK).

I couldn't reproduce, but probably this also happens with calculateRowsValues.

A major purpose of demos is to act as howtos to others. Want docs on how to do this.

Goal is to maybe integrate some of the features of http://www.dondevanmisimpuestos.es/ccaa/ as embeddable openspending widgets.

I'm opening this issue as a place to discuss how we evolve openspendingjs going forward.

Current Situation

This is my rough understanding and is likely wrong - please correct and amend!

README states

Visualisations and other utils that use data from OpenSpending to help people understand where and how money is being spent.

Note in terms of file structure:

We are currently in a migration stage in order to make contributions easier and more consistent, so the code base might be a little confusing. The best place to add your visualisation is src/. We're in the process of moving all of our visualisations and utils into that folder (to make the build process cleaner and simpler).

External libraries (requirements) are placed in lib/vendor/. There are a lot of files there at the moment. Some of those files are old versions of these libraries. We're in the process of cleaning these files up, removing unused files.

So, basically we have:

• src/utils - common core code
• src/visualisations - visualisations

Sort of deprecated:

• lib Common core code such as JS that wraps the OpenSpending API
• app - various viz / apps (see below)

apps:

bob  # the query builder (?)
browser # data browser (?)
bubblemap # bubblemap viz
dailybread # dailybread viz
data_table # data_table browser
dimensions-editor # dimensions editor (OS import process) (cf http://nickstenning.github.io/openspending.modeleditor/)
faceter # not sure 
spending-map # not sure but maybe the front page map on OS.org
treetable # not sure ...

The Future

Here's a proposal for discussion:

• Core JS library (no visualizations) lives in openspending.js (could keep openspendingjs name but rename might be useful to symbolize change)
• Every visualization gets its own separate repo named e.g. openspending.{vizname}.js

Lastly some process points:

• We should complete clean-up of this repo asap (possibly in tandem with doing the split-out)
• Anyone can develop a new viz but for it to live in openspending orgnanization it needs to "approved" (i.e. checked for quality, documentation, and that there is an ongoing maintainer).
• We document the set of visualizations people can use off-the-shelf somewhere prominent

Why?

• Core code is more clearly demarcated from apps/viz and can be maintained more easily going forward. It also can become a core dependency that people can more easily use in their own work without getting caught up in viz stuff
• By splitting out visualizations we make them easier to work on and more clearly distinct. Dependencies are also clearer etc.

Moved from openspending/openspending#665

Have a build process for building openspendingjs into one consolidated file.

Email from @tryggvib

http://lists.okfn.org/pipermail/openspending-dev/2013-May/000791.html

I've taken a stab at this and created a build process for openspendingjs
using Gruntjs (http://gruntjs.com). To make the build process clean I'm
working from a new directory structure (that sits besides the current
one). What I envision we will have in the end is something like many of
the open source / javascript libraries out there:

openspendingjs
    |
    |- dist/ (where build results go so we'll have versioning, examples below)
    |     |- openspending-.js (concatenated javascript files)
    |     |- openspending-.min.js (minified javascript files)
    |     |- openspending-.css (concatenated css files)
    |     `- openspending-.min.css (minified css files)
    |
    |- lib/ (where we put external libraries, examples below)
    |    |- kartograph-.js
    |    `- raphael-.js
    |
    |- src/
    |     |
    |     |- utils (various openspendingjs utils, examples below)
    |     |     |- openspending.amounts.js (utils for dealing with amounts)
    |     |     |- openspending.colors.js (utils for managing colors)
    |     |     |- openspending.aggregator.js (API aggregator)
    |     |     `- README.md (information about the utils)
    |     |
    |     |- visualisations (where visualisation files go, examples below)
    |     |     |- jquery.choropleth.js (jquery plugin to create a choropleth map)
    |     |     |- jquery.treemap.js (jquery plugin to create a treemap)
    |     |     `- README.md (information about the visualisations)
    |     |
    |     `- css (where stylesheets go, example below)
    |           `- treemap.css
    |
    |- package.json (information about the package)
    |- README.md (information about the project)
    |- CONTRIBUTORS (who has contributued)
    |- LICENSE (license for the project)
    `- Gruntfile.js (build configurations)

Like I said, I've taken a stab at this and created a branch with this
structure (without touching the other structure - I even still use
lib/vendor/ instead of lib). I deleted some things (like the flash
folder). Here's the branch on github if you want to take a look:
https://github.com/openspending/openspendingjs/tree/build-process

There are some new things there like how the visualisations are managed.
I moved them into being jquery plugins. They don't have to be, we could
use our OpenSpending object, but I did this when I felt visualisations
shouldn't be a core part of OpenSpending (I felt the utils should be).
So I thought having them as jquery plugins might maybe result in more
visualisation contributions (in retrospect I think I should move them
back under OpenSpending so we'll be famous!). However it is cool that in
order to create a treemap on a site one can now either do:

<div class="treemap" data-dataset="uppgjor-rikissjods"></div>

or:

$(some selector).treemap({data:{dataset:'uppgjor-rikissjods'}})

Plus a bunch of other options (it just feels natural when you've worked
a lot with bootstrap). Of course you need to load the
openspending-.min.js and openspending-.min.css files.
This approach might make a plugin creation (e.g. for WordPress) a lot
simpler (drop in the js and the css files, make a simple parser with the
configurations to create a new div).

I'm really interested in hearing your opinions. I've probably made some
weird decisions here and there while trying to keep all of those balls
up in the air, but this is how I think the openspendingjs library could be.

I'm especially interested in hearing from visualisation experts out
there. Would this make contributing new and fresh visualisations easier?

Moved from openspending/openspending#675

A gallery browser could replace the function of the dataset browser, and be implemented with some of the filtering functions as at Datawrapper: http://datawrapper.de/gallery/#

This could make browsing of OS data more appealing to users, if we assume that users prefer visualisations over names of datasets.

• Look nice
• Support show / hide (rather than just hide)
• Allow for choosing year (and possibly other 'cuts')
• Provide help (?)

Our Treemap widget sends its element's id to JIT in https://github.com/openspending/openspendingjs/blob/master/widgets/treemap/main.js#L180. If the element doesn't have an id, this line will break.

I'm not sure if JIT requires an id for its widgets but, if not, it would be useful to remove this requirement in our side.

Here is @vitorbaptista's work on this from earlier this year:
https://github.com/openspending/openspendingjs/tree/feature/time-series

On the bar chart idea I would add CIVIO's project as well:
http://www.dondevanmisimpuestos.es/ccaa/

Question:

• What is the ideal model?
• How many days are left for completing this thing for either the WordPress plugin or OpenSpendng core?

Comments @vitorbaptista @tryggvib ?

Look into Karma as a test framework for openspendingjs. It should integrate well with grunt.

Generate a treemap with:

<div class="treemap" data-dataset="haushalt-muenster-2009-2012" data-drilldowns="bereich" data-year="2012">

and it doesn't draw every item (it only draws 5 out of 22). It looks like JIT computes height and width to negative values for entries 6-22.

Moving from openspending/openspending#31

The Compar-o-tron B helps users to discover and share the proportions between different classes of government spending. This satisfies the need to contextualize individual items, policies or functions of government by contrasting them with one or many more tanglible other items.

Images

A first mockup done for original WDMMG project. One might do things quite differently for this version ...

Functionality

• The Compare-o-tron helps users to find spending classes by filling a shopping cart.
  • This can be filled either while browsing the site in normal mode (i.e. via treemaps, bubblecharts) or through a direct search function.
  • Search results may already be presented as bubbles. Search should have auto-complete.
  • Care should be taken to warn the user if they attempt to include nested items (e.g. "Defense", "200 new Tanks" - which is part of the Defense budget).
  • The selected items can include: entries, classifiers, entities. Each of these needs to be assigned a unique, "typical" value, e.g. for "Defense" in COFOG this could be "annual expenditure in 20xx", for "Mineral Oil Taxes" in the German FKP it could be "annual projected revenue for the current year".
  • The selected items should also have color and icon specifications associated.
• After selection a few items, these will be presented in a joint graph as bubbles.
  • Each compare-o-tron receives a unique URL, either by saving the cart to the DB or by URL-encoding it.
  • The user creating the compare-o-tron may also want to comment on the diagram.
• A gallery of existing compare-o-trons is available.
• Compare-o-tron can be embedded into another article, e.g. a news story.

Implementation

The key features for compare-o-tron are:

• Named items (things that can be added to the cart)
• The cart
• The ability to display non-related bubbles.

The following would be one option to implement the back-end for named items:

1 Introduce another section in the mapping with the following format:

{ 
    "named_items": {
        "cube_query": { ... },
        "name_key": "some_dimension.label",
        "color_key": "some_dimension.label",
        "weight_factor": 0.5, # will be used to scale volume.
        "default_color": "..."
    }
}

(rinse, repeat for icons if needed)

1. Add an extension point to generate_views in the loader
2. Hook into this and generate a db.dataset_name.named_items collection with the named attributes and amount from the drilldown.
3. Query this collection via a custom controller.
4. Visualize as bubbles.

More Information

• page in wiki: http://wiki.openspending.org/Visualization_Comparotron (not accessible any longer)
  • Mockup: see attached image.
    • RP: items on left side are results of search so far. You can drag these across to right. Should then be able to hide search list and share with friends.
    • RP: note this is mockup and does not have to determine final design

Moved from openspending/openspending#161

(requested by CSH in Germany)

See original issue for comments

Page: http://jmblog.github.io/openspending-icons/
Code: https://github.com/jmblog/openspending-icons/

Moved from openspending/openspending#500

Introduce a widget for bar charts which can display various drilldowns, especially developments over time.

Constraints:

• Must work in IE
• Must scale to display anywhere between 3 and 30 bars
• Axis labels etc. must be set up correctly from metadata

Optional:

• Logarithmic scale
• Support both bar and column charts.

See original issue for comments

Consider the treemap in http://slovakia.openspending.org/municipality/#piestany/year:2012. In 2012, the element Školstvo is the second biggest, and it's colored pink. If you change to 2011, it becomes the biggest, and turns red.

With this color change, it's really hard to keep track of its changes. It would be great if each element had always the same color.

To do this, you'll have to change https://github.com/openspending/openspendingjs/blob/master/widgets/treemap/main.js#L164-L170.

See app/explorer-v2/

Features:

• Bubbletree (already largely there but could remove option to choose breakdowns -- configure it directly)
  • Get it working again
• URL manager so url hash updates and is used on load

@markbrough has already done most of this.

Moved from openspending/openspending#157

Original text:

We should have a simple way of presenting a balance sheet, i.e. a simple comparison of individual items with a dimension that divides between Equity, Liabilities and Assets.

Simple example: http://mirm.eu/privat/kommunalfinanzen/index.php

Take a look at original version for comments

Moved from openspending/openspending#563

The BubbleTree vis requires a container with height > 0 otherwise it will break.
Of course it's nice to configure the BubbleTree size via the container, but in this case the container is potentially also used for the DataTable vis so the height cannot be fix (e.g. height: 400px).

The current fix is having min-height: 400px but the data table is then sometimes bigger than necessary. I think the right fix would be that the BubbleTree lib defines a default height if container height is zero.

See original issue for comments

Check http://www.webpagescreenshot.info/img/900698-1022201270603PM

Would be preferable to open visualisation links in parent or blank window.

Also don't round tops - it just looks ugly :/

AjaxSolr is great but rather heavy duty and does everything itself (UI handling etc). Suggest switch to FacetView library: https://github.com/okfn/facetview

https://github.com/openspending/openspendingjs/blob/master/lib/utils/utils.js#L169

!?

Moved from openspending/openspending#483

From Miro Ščibrany,

• TreeMap:
  • I would appreciate displaying time period, filter conditions and Total amount
  • I would appreciate a radio button to switch between display absolute values / percentage against the total

See original issue for comments

boot.js calls JQuery.sub() in https://github.com/openspending/openspendingjs/blob/master/lib/boot.js#L11-L12. This was deprecated in 1.7, and removed in 1.9. I've changed this call to simply JQuery and the Treemap worked fine. Not sure if something else breaks.

@nickstenning Do you remember why you added it?

The dailybread is heavily tied to the UK and needs to allow for configurations for other countries.

This way of detecting IE is deprecated since JQuery 1.3, and removed in 1.9 (http://api.jquery.com/jQuery.browser/). The way to go now is using feature detection.

Moved from openspending/openspending#501

Introduce a widget for line graphs which can display various drilldowns, especially developments over time.

Constraints:

• Must work in IE
• Must scale to display anywhere between 3 and 30 bars
• Axis labels etc. must be set up correctly from metadata

Optional:

• Logarithmic scale
• Stacked area line graphs

See original issue for comments

Similar to #46, there is not a way to move up a drilldown level in bar charts.

Moved from openspending/openspending#643

/cc @vitorbaptista
Couldn't find the files to make a pull request, so I'm opening an issue here instead.

There's a missing font on the embeddable iframe:
treemap.css#16: font-family: 'OpenSansRegular';

This could be easily fixed by adding the font to the iframe.

As a good practice I would also recommend adding a generic sans-serif font to the stack, making it less shallow and avoiding a 'Times New Roman' as default, which would make the code look something like this: font-family: 'OpenSansRegular', sans-serif;

Also I think it would be awesome if I could add an array of custom colors to the Treemap, since the current ones are destroying my layout. 😢
The call for the color array is at main.js#163.

Checkout original issue for comment

A gallery of OpenSpending visualisations would be incredibly helpful to people developing satellite sites. Also it would help to give a shout out to contributors of OpenSpendingjs.

Could be done via github pages.

(aka. openspending.github.org/openspendingjs/visualisations)

Moved from openspending/openspending#509

For a dataset with a geographic dimension, the user should be able to aggregate values over that dimension and then use the aggregated data to display a choropleth map.

The hard aspect of this is having the "right" map, i.e. one that captures all the values in the dataset but does not extend much beyond it. Even worse, the identifiers used in the map for regions must also be found in the data, or an alignment must be made at run-time.

In an ideal case, this would also work for subsets of the data and display an appropriate map to match each query's result space.

Take a look at original issue for comments

Factor out table section of spend browser into standalone widget named TableView so it can be reused.

Drilldown behaviour is fixed to expand the clicked element (drilldown into the data). Users might want something else to happen and should be able to change the behaviour.

• Integrate TableView widget (depends #2)

Moved from openspending/openspending#450

We should find a single method here, if possible one that is compatible with transifex.

See original issue for comments

Moved from openspending/openspending#641

Hello there. I don't know if this is a known issue but there's a bug when trying to download data generated by a "Table of aggregates" visualization, the "Data" button [last one in the bottom right of the table] isn't working.

By looking at the console I have an error at the getDownloadURL function [line 53 of main.js] where the var data is undefined.

Moved from openspending/openspending#494

Unless I am mistaken, there is not a way to go up one level with treemaps. Possible to put sth intuitive in?

See original issue for comments

Where can i customize the Bubblemap Labels and Title label? How to enter Metadata? in which format?

I have my CSV file, here https://github.com/RCGTDev/A-donde-Van-Mis-Impuestos/blob/gh-pages/csv/by-cofog1-then-cofog2.csv

How can I visualize the labels of cofog_level1_name and cofog_level2_name?

Moved from openspending/openspending#664

Background

http://lists.okfn.org/pipermail/openspending-dev/2013-May/000783.html

I've been looking at openspendingjs and trying to wrap my head around
it, with the intention of setting up a build process for it and allow
people to contribute new visualisations.

I think openspendingjs is in serious need of some refactoring and a
clean-up. If this is something we want people to contribute to we have
to make it easy for them to do so. In it's current state it's
under-documented, has some weird stuff in it, is tightly coupled with
specific sites, and has a confusing hierarchy.

Here are a few things I noticed as I randomly browsed through:

• 'flash' directory in openspending_js_ with a flash implementation of bubbletree (is it used?)
• 'app/datacache/' has json files that aren't used in openspendingjs (e.g. cofog1-cofog2.json)
• 'app/deficit-and-debt/' only includes an html file with some inline scripting
• What's the difference between 'app' and 'widgets' (site-specific vs. general)?
• Does openspendingjs really use all the libraries in 'lib/vendor/' (I haven't checked)?
• 'widgets/treemap/js' seems to include it's own thejit (javascript infovis toolkit) library
• Result of the build script isn't obvious (spits out 'prod/boot.js')

I'm not looking for a bikeshed discussion and I'm not pointing fingers.
I'm just raising some things I noticed to show what I mean by
refactoring and clean-up. This should help us get started on discussing
openspendingjs. Here are some questions:

• What is it we want openspendingjs to do? ( http://www.ur1.ca/e2g62 ?)
• Can we make the project structure simpler/readable? (or at least document it?)
• Will it break anything (external) if we refactor/clean-up openspendingjs?
• Can we make it easy for visualisation creators to add visualisations to openspendingjs?
• Is there somebody who would like to help us refactor/clean-up openspendingjs?

I don't think cleaning openspendingjs up is a huge task. It would make
the build process a lot easier (not having to deal with a lot of cruft)
and more importantly make it a lot easier for people to contribute
visualisations that use data in openspending.org

We no longer need this directory in master (long obsolete and unused). We do want to have a historical copy but that's in the repo history!

I'm happy to do this!

At the moment, instantiation and state serialisation for view widgets (explorer, treemap, bubbles) is inconsistent, which means hard-coding custom to each visualisation in OpenSpending (see okfn/openspending.plugin.treemap for an example).

We should have a front-end interface class defining how we expect view widgets to behave.

Current proposal:

class OpenSpending.IViewWidget
  # Parameters:
  #   element: a DOM node in which to instantiate the widget
  #   context: a JSON object defining the OpenSpending context (see details below) [optional]
  #   state: a JSON serialisation of the state of the widget
  constructor: (element, context={}, state={}) ->
    throw "Not Implemented"

  # Serialize the current widget state to a JSON object
  serialize: () ->
    throw "Not Implemented"

The state parameter to the constructor is entirely "schemaless", and is simply whatever the serialize method spits out.

The exact format of the context parameter remains to be decided upon, but should probably provide some information to the widget about, e.g. which dataset is being viewed, which dimension page the user is on, and so on.

If I add a new directory fooin /widgets/and copy and paste the widget files from another widget (say AggregateTable) to that folder, the new widget won't appear in the visualization selection menu. Is this because the widgets metadata files are parsed on server side?

How can I as frontend dev create and test a new widget?

thx

I just tried to use _.pick which is only available in underscore >1.3.3. There is already 1.4.4 and openspendingjs comes with with 1.3.3.

We should upgrade dependencies.

I think it's about time for us to integrate maps into OpenSpending. Some datasets already have a geographic dimension, like ukgov-finances-cra has the dimension region. Now the explorer need to know just two things: does this dataset have a geographic dimension and is there a map prepared for it. We could even reduce this check to a simple check for existing files:

Let's say we have a folder maps/somewhere. For the uk example we would create a folder maps/ukgov-finances-cra/ and put a map source inside, named after the region dimension: region.svg. For datasets with more geographical dimensions, like eu-commission-fts which has dims for three nuts region levels, we could place different maps into that folder.

On the frontend side we would eventually display a link show map to link to the map view. At the map view, a dropdown would allow the users to select the geographic dimension if more maps are available.

The creation of the map sources would become a manual integration step. Although I'm already working on a tool to create svg maps, there's always the risk of mismatching regions. However, in many scenarios this integration is a rather simple task.

For the rendering side, I've written a JS library that renders the svg maps, too. It comes with support for some basic datavis features, like showing maps with bubbles or pie charts on it or coloring regions. OpenSpending would be a perfect use case for this lib.

Btw, here's a running demo of such a map: http://vis.okfnlabs.org/ftsdemo/?level=nuts2

What do you think?

A lot of people want the breadcrumbs so they should be the default behaviour. Without them the treemap becomes a dead end traversal visualisation.

Currently the apps integrate their jQuery lib directly from Google. Also, they all use different versions (1.4, 1.4.3, 1.6, 1.7.1).

It's not very likely that people already have all those versions in cache, so the CDN benefit is negligible. Moreover, there is a privacy impact since most visitors will send a referrer to the Google CDN.

Directly integrating jQuery (preferrably only one version) would rectify this and make openspending sites more independent from third party network stability.