dataviva / dataviva-site

Rapid data visualization engine

DataViva

A Collaborative Planning Tool. dataviva.info

Support

DataViva will run on all modern browsers so long as they have Javascript turned on and full support for SVG graphics. This includes the latest versions of Firefox, Chrome (Chromium), Safari (WebKit), Opera and IE.

Note: Internet Explorer versions 8 and below will not work as they do not have SVG support built in.

Installation

DataViva is a web platform built using Flask, an open source Python library for interacting with HTTP. this installation guide is written assuming a Linux or Linux-like environment. The following software is required to be installed locally in order to get DataViva running:

• Python
• MySQL
• Redis (at least version 2.8.6)

1. Clone from github (this will create dataviva folder in the current directory)

   git clone https://github.com/alexandersimoes/dataviva.git
   

2. [optional] Create a virtual environment. We suggest installing virtualenv with virtualenvwrapper especially if the machine you are using is used for many other web projects. This allows python libraries to be installed easily and specifically on a per proeject basis.

   Once this is complete, run the following to initialize your dataviva environment.

   mkvirtualenv dataviva
   workon dataviva
   

3. Install the required Python libraries

   pip install -r requirements.txt
   

4. Install librsvg

   apt-get install librsvg2-bin (Ubuntu)
   brew install librsvg (Mac)
   

5. Create a MySQL database and import the latest database dump

   # Import the schema
   gzip -dc DV2015_DB/*/schema/*.sql.gz | mysql -u USERNAME -p -h 127.0.0.1 DB_NAME
   # Import the content
   gzip -dc DV2015_DB/*/content/*.sql.gz | mysql -u USERNAME -p -h 127.0.0.1 DB_NAME
   

6. Set the following environment variables

   (if using virtualenv) add the following to to the bottom of your virtualenv activate file (virtualenv_root/bin/activate).

   export DATAVIVA_SECRET_KEY=my-super-secret-k3y
   export DATAVIVA_DB_NAME=dataviva
   export DATAVIVA_DB_USER=root
   export DATAVIVA_DB_PW=my-pw
   export DATAVIVA_OAUTH_GOOGLE_ID=
   export DATAVIVA_OAUTH_GOOGLE_SECRET=
   export DATAVIVA_OAUTH_TWITTER_ID=
   export DATAVIVA_OAUTH_TWITTER_SECRET=
   export DATAVIVA_OAUTH_FACEBOOK_ID=
   export DATAVIVA_OAUTH_FACEBOOK_SECRET=
   * export DATAVIVA_ACCOUNTS=True
   export DATAVIVA_API_BASE_URL=https://api.staging.dataviva.info/
   
   * only necessary if you want to enable user accounts
   

7. Handle en/pt subdomains for translations. For example, when testing locally you may add these lines to /etc/hosts

    127.0.0.1 en.localhost
    127.0.0.1 pt.localhost
   

8. Run the site locally!

    python run.py runserver
   

dataviva-site's Issues

Better Screen Resizing

Pages should only resize when they are larger than a certain size (say, 900px). This would help considerably with mobile viewing.

Ability for Admins to Edit/Remove User Questions

We have some users that does not put a question, or use it too long or too short.

So we would like to be able to edit questions, so we can fix some users errors.

A example of question missing a title:
http://dataviva.info/about/question/55_/

Other thing is that, sometimes when a user use a space, it shows a   instead.

What you think about use question to be editable and to strip html?

Find a way to show data download progress

Currently, we are not able to show data download progress when building an app because we do not get sent progress for GZIP transfers.

We need to GZIP the data, because it reduces the file size immensely, but because of this we are not able to show progress. The d3.json connection does not currently allow us to estimate the overall file size and progress of GZIP files downloading.

https://github.com/mbostock/d3/wiki/Requests#on

Mid-level depth selector in App Builder

Now that the data has been aggregated at each level, we can expand the depth selector in the apps to show mid-level depths.

Filter a Location in GeoMap

When we use the Geo Map App and the filter a specific city, it shows the earth globe.

Nominal Growth Rates

We have some growth rates that are nominal, means that we do not fix values considering the inflation.

So the names should be changed to Nominal.

Example:

• Annual Wage Growth Rate -> Nominal Annual Wage Growth Rate
• Taxa de Crescimento dos Salários Anual -> Taxa Nominal de Crescimento de Salários

Should be applyied to others growth rates.

Color Selector Buttons by Item Color

Could be interesting to have the "Select" button be colored in the specific item's color. Currently, the only thing that references an item's color is it's icon and title, as in this current screenshot:

If we were to color the "select" and "classes" buttons to match the item's color, it could give a better visual connection between the two.

Optimize "make_query" function

Some specific data queries, for example, when specifying location, industry, AND occupation (rais_ybio table) take longer than others.

Here is the function that needs to be optimized: make_query

Wrong Average Monthly Wage for Location With Distances

Ff we take the total wage and divide by total number of empoleeyes.. will we get the avg monthly wage?

http://dataviva.info/apps/builder/rings/rais/mg100404.30/c2592/all/isic/?controls=true&year=2012&color_var=color

If you see at this link we have total of $ 1,55 and 651 employees so. the avg should be $ 2.380 , but you can see that dataviva is showing 1.818

we did some investigation here.. and looks like that.. if you use 30 km distance from location.. it get wrong number

see this link (withtou 30km range) it shows a good number

http://dataviva.info/apps/builder/rings/rais/mg100404/c2592/all/isic/?color_var=color&controls=true&year=2012

Talking to Alex he said that could have to do with our aggregations when we do the 30km call from the server there is no entry in the database for this value, we need to aggregate values and we could be summing avg monthly wage, instead of recalculating it

Path Changed for WTF Validators

Using Flask-WTF 0.9.2 we have some problems regarding the path of imports for WTF Validators, like Required, Length ...

Changing for the example above worked for older and newer versions, but we sould take a look if it will keep using the same sources.

from wtforms.validators import Required,Length,url

"Star an App" always asks users to log in

When trying to star an App, it asks to loggin. After loggin, when try to star again, it asks to login.

Even if you are logged and try to star an app, it asks to log again.

Take this link as a example:

http://www.dataviva.info/profiles/cbo/7911/?app=1

Sometimes first related app does not work

http://dataviva.info/apps/embed/tree_map/rais/all/g4771/2410/bra/?depth=bra_8&value_var=num_emp&color_var=color&controls=true&year=2005

MesoRegion with Wrong Name

http://dataviva.info/apps/builder/tree_map/rais/mg/all/all/bra/?controls=true&year=2002&value_var=num_est&depth=bra_4&color_var=color

The name of the Meso is showing "Triângulo Mineiro / Alto Paranáiba" but should be "Triângulo Mineiro / Alto Paranaíba"

API Docs

If a developer or a user would like to make some dynamic views using Dataviva data, is possible using the json requests. But there isnt a doc explaining how to use that, considering filters and results.

Product Complexity Missing

There is some products that does not have a complexity number associated to.

There are around 121 products without complexity, so the ideia here is to find a way to calculte this number for those products.

Better Error Page Design w/ Error Log e-mailed to Admins

We do not have a default page for error, so users is viewing the stacktrace and debug instead of a error message.

Other good thing to do is to send a email alerting us about the error, so we should be pro active considering bug fixes.

Embed Footer Message

Would be nice if we have a message in footer more attractive to users, taking the example of pantheon.

Example:
"Explore the interactive on the Pantheon website."

Different values for total Exports

If we see the ring's for product called "Estanho" in Brazil, it shows a total of 132 million USD. But if we see the same app for Roraima State, it shows the same value.

Roraime does not export Estanho, and maybe Dataviva is taking BRL export value in case of a region that does not export.

Use D3plus combo toggle for "color" UI

When changing colors for an app, the "Growth" values should be separated out to a separate toggle, that only appears when Growth is selected.

Product x Activity

Would be interesting to show to the user how we made the map between Products and Activities classification.

Maybe a static table in Learn More, and linking in some other places in doc should be enough to help users.

Guide Page Icons

Could be interesting to have each Guide Page have the icon associated with it (from the button the user clicked in the previous screen) in the left box, similar to the icons/flags in the profile screen.

Wage Average Nesting is Incorrect

When d3plus is calculating nesting levels for average wage, it is averaging the average, and not recalculating it based on summing wage and num emp. Once d3plus is updated to support passing functions for aggregation, we can fix this issue.

Guide Breaks when Search for a 4-digit HS Product

another bug
In guide is possible to select a specific product.

Navigating over products, the app filter to show just 4 digit HS products.

But when you search for a 2 Digit HS and select the product, we will have a error in the next steps.

As we know, should be not possible to select 2 HS Digit, so maybe should have a param that can filter from the search HS that has digit lower than a specific number

We talked to Alex and he said: yesss supposed to be doing that... strange that got through ok good to know

To reproduce the bug, go to the link:
http://dataviva.info/guide/hs/select/

Select PT and search for: Mobiliario
it shows a product with 2 digit HS, so.. if you selec this product.. it breaks the next steps..

Redis type error

Running the system on ubuntu 12.04 with redis 2.2.12 this script return:

TypeError
TypeError: an integer is required

The error is in line 212 with this code:

self.redis.setex(self.prefix + session.sid, val,
                         int(redis_exp.total_seconds()))

Average Monthly Wage shows different values

You can see in these printscreens that the values for monthly wage are different in Geomap comparing to Treemap. Looks like that the Treemap value is wrong cause is different from nacional average.

PS: Looks like that problem about making a average of averages...

Social/Sharing Meta Tags

Please,

add some meta tags to a better integration with facebook.

http://davidwalsh.name/facebook-meta-tags

secex_ypw table aggregations not correct

select sum(val_usd) from secex_ybp where year = 2012 and hs_id = "158001" and length(bra_id) = 8

select sum(val_usd) from secex_ypw where year = 2012 and hs_id = "158001" and length(wld_id) = 5

Problem sharing on IE10

Liability

We need some term of liability on the site like: we will make sure the developed platform presents the data provided accurately, but shall not be responsible for controversies emerging from the interpretation of that data. Maybe we can add to the terms of use.

Format Data When Exporting Data

Would be better if we have a validation that:

• When exporting in English shows values using dots (1,234.00)
• When exporting in Portugues, shows values using comma (1.234,00)

Another topic is to use UTF8 as a encoding, cause when exports a XLS and opens in Windows Excel, we have problems with special chars.

• There is many DownloadForm but all references points to just one (dataviva.data.forms) , so remove others.
• There is some import of DownloadForm not using (remove them)
• Look at apps.download and utils.make_query (download if)

See the example:

Problem Exporting PDF

There is some cases that is not possible to export data from the apps.

The problem is not to download a CSV file, but when export to PDF, it shows the message:

413 Request Entity Too Large

If you go to the link above you will be able to reproduce the problem.

http://www.dataviva.info/apps/builder/tree_map/secex/all/all/all/bra/?depth=bra_8&value_var=val_usd&color_var=color&controls=true&year=2012

We have some possible solutions here, like lock the export when the data is about a huge dataset (maybe for states vision) or try to increase the request limit.

Regards,

Syntax error

In dataviva/init.py has a syntax error on line 11, see:

from flaskext.babel import Babel

The correct is

from flask.ext.babel import Babel

Distance, Opp Gain and RCA not showing for products with complexity

Some products do not have complexity, same in atlas.

But in case that have a complexity it must have:

• Distance
• Oportunity gain

So attached here is a example that shows distance and complexity but no Opp Gain.

In case that is not exporting data, show RCA with 0 value and a link to glossary explaining why that hapens.

HTML Chars in Ask Forms

When a user makes a question it can use enter or other special char that makes appears a HTML tag inside the question. LIke this example:
 

Here a true example:
Can one use any type of data  in this application. e.g. social media?

This erros shows just in list questions, to see it search for "social media" in
http://www.dataviva.info/about/contact/

Show details do not show the error:
http://www.dataviva.info/about/question/is_this_to_be_a_canned_app/

Sometimes a user inse

Special Char in Ask Form

When using special char in que title of a ask it gives a error.

Seems that the problem is related to the filter, when using str()

• File "/web/dataviva.info/dataviva/about/views.py", line 88, in ask
_body = filter.clean(str(form.body.data))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe7' in position 10: ordinal not in range(128)

Update Homepage Video with Content from Escritorio

Ranking Dinamic Grid

When looking at ranking pages we can see that the size of the window is dinamic.

So when the number of columns of the tables increases/decreases it changes the size.

Some users are getting confused, so looks like that is bad for usability.

What u think about making that static?

Take a look at this example:
http://www.dataviva.info/rankings/2012/bra/2/?order=val_usd.desc

Share always in English

The share link uses the title always in english, not respecting PT language.

Occugrid tooltip improvements

Dave and Alex,

I was taking a look at the occugrid and found some regions that have RCA>1 but have not filled areas. You said that when the location had RCA it would show all filled because was compared to itself. What is happening?

Remove "beta" from site Logo

The new logo file is ready, and will be switched in as the final change before the deadline.

Better App Error Messages

The site should differentiate between when there is no data for a specific selection and when the app simply cannot be built with that selection. For example, Occugrids and Scatters cannot be built for Brazil (only for locations inside Brazil). Those apps should show a different error message.

Planning Region - Wrong numbers?

We did some analysis about RCA in different kinds of region: Municipality, Meso, Planning Regions and States.

We could see that for Municipality, Meso and States the numbers looks good, but for Planning Regions looks like that is a problem.

If we take the example of Rio Doce Planning Region x Vale Rio Doce Mesoregion its a good way to check cause these two has the same municipalities.

We attached 2 files, one for Planning and other for Meso and is possible to see that the numbers are different, but should be the same.

Other thing is that the products shown in Planning is less than for Meso, take a look at HS 052507, it can be found at Meso but not at Planning. Maybe this is a tip showing where the bug can be..

Link Definitions to Glossary

Would be great if the definitions within the App tooltips linked to the full definition located in the Glossary section of Learn More.

This functionality needs to be built into D3plus, and is currently on my short list of features to implement. This will definitely be ready for DataViva 2.0.

"No Data Available" not in Portuguese

Some javascript text is not translated.
The most common example is "No Data Available" , when showing a chart in d3plus without data.

Rings "false" Node Title

Sometimes in Rings, certain primary connection nodes have "false" for their title. See this example: http://dataviva.info/apps/builder/rings/rais/mg/m7110/all/isic/

Keyboard functionality in attribute selector

It would be nice to add keyboard functionality to the filter selector, for example:

• up/down arrows to highlight different items in the list
• enter/return to select a specific item
• esc to go back 1 level

Profile selector breaks when scrolling page beneath

Steps to recreate:

• Navigate to a Profile page (such as this page)
• Click on the page title to open the Selector window
• Scroll past the bottom of the list in the Selector window

You will see the page beneath the Selector begin to scroll (which shouldn't happen), and the "back" button in the selector becomes unclickable.

Download Map Image

When trying to download the map image, we got a error:

413 Request Entity Too Large

App Help Sizing on Tall Screens

Currently, the images/videos associated with the app help (found by clicking the question mark button on the /apps/ page) display incorrectly when the aspect ratio of the window is < 1.

Show N/A in tooltip for certain missing indicators

Sometimes is normal to not have some indicators, like complexity for products that is not in Atlas.
And will no have distance and opportunity gain for those as well.

In this cases would be better to show "Complexity: Not Available (click here to know more)" and then link to a page in glossary explain why some products do no have this info.