okfn / dashboard

Dashboard for communities: what's up, who's here and what are they working on. Designed for use at the Open Knowledge Foundation.

A Community Dashboard for the Open Knowledge Foundation. See wiki for details of project: http://wiki.okfn.org/Community_Dashboard.

• Live site: http://okfnlabs.org/dashboard/
• Git repo: http://github.com/okfn/dashboard
• Issues: http://github.com/okfn/dashboard/issues
• Mailing list: http://lists.okfn.org/mailman/listinfo/okfn-labs

Developers

Deployment is via gh-pages but you need to have "built" the app using brunch first. To make changes and deploy you do:

1. Edit stuff in master branch and commit
2. Run brunch build this will create a public directory
3. Switch to gh-pages
4. Copy contents of new public folder to root directory
5. Commit and deploy gh-pages branch

dashboard's Issues

Latest Tweet, Latest Facebook post, Latest blog post

A small box for each of these would be useful to make sure that we're updating everything every day! No need to have detailed info or even pics, just last update and a brief bit of text pulled in.

C'est possible?

Make gather scripts idempotent (no duplication into activity table)

Ensure we don't duplicate items in the activity table (do we need a test for this)?

Backend storage infrastructure

Need somewhere to store data. Plan is to webstore. Need to:

• Set up user and db (suggest okfn and dashboard)
• Have some basic config and code for connecting using webstore client

Get member data from http://okfn.org/ and put in person table

[super] Switch to Recline as default Data Viewer

• Members map (need geo support in Recline)
• Activity stats

Graph showing OKFN membership trends over time

Member / Persons should have a 'created' or similar attribute that would allow us to compute this.

Create full list of user stories

So that we can prioritize, plan next steps (and get something launched asap) we should create a set of User Stories.

User stories google doc here

Track mailchimp lists

Mainly OKFN announce.

use the leafletmarkercluster to reduce the number of point on the map

I think that including this leaflet function can be useful to the performance and to reading of the map
More info here
https://github.com/danzel/Leaflet.markercluster#leafletmarkercluster

Update gather scripts to reference person in person table

Could do this post-hoc as part of a reconciliation step rather than on load (probably better).

Rework members map

• Run off webstore person table
• Can not have 1Mb openlayers file
• Convert into backbone view and integrate into overall app

Reconcile each activity to a project

Each activity should be assigned to one and only one project.

Scrape mailman stats since September

• Fortunately, mailman keeps a log of new subscribers
• Unfortunately, the daily scraper job appears to be crashing

Rearchitect server cron jobs

At present, Wordpress front end runs a cron task, ~/cronjob_list_members.sh. This connects to the remote DB server, pulls the members list, and uses regexes to hack it into a CSV file which it places in a public directory.

Problems:

• Piping SQL results through sed to create a CSV? Really?
• We've discovered any file made public on the wp server (eg. http://okfn.org/dashboard/file.csv) will also appear on all sites in the multisite (eg http://openglam.net/dashboard/file.csv), making a mess.

Suggest the cron job runs on the server, using SELECT INTO OUTFILE to avoid the sed mess, and uploads it somewhere public. For example it could push it to a repo on GitHub.

The Dashboard gather script just needs a public up-to-date CSV file of members.

Display activity and info on a single service (US#8)

See user story #8

Decide on top priorities

This should be a (list) of user stories in order

Work Order

• US4 - Basic stats and trends for projects/activities
  • US3 - activity stream per project
  • US5 - what projects are there
• US2 - Pan-OKFN activity stream
• US1 - Member directory and map - but not so sure ...

Priority Order

• US4 - Basic stats and trends for projects/activities
• US2 - Pan-OKFN activity stream
• US3 - activity stream per project
• US5 - activity stream per project
• US1 - Member directory and map - but not so sure ...

RP would vote for (very rough order)

• US2 - Pan-OKFN activity stream
• US4 - Basic stats and trends for projects/activities
• US1 - Member directory and map - but not so sure ...

Try-out js: simple person search and display page

Aim: create a page that allows you to search for persons with results shown in a list on LHS pane. Selecting a person should show full details on RHS pane.

Details:

• Create a an html page person.html in app/
• Javascript and css should go in app/js and app/css
• Have simple form for searching at top of page. Intercept submit in javascript and do a search with it
• Use jquery ajax jsonp call to get list of people from http://webstore.thedatahub.org/okfn/dashboarddev/person?name={name}
• Display these people in html list. Make name in html link
• Click on html link should bring up details on right

After we have done basic version next step will be to refactor to use Backbone so worth starting to read up on this now.

Automate extraction of member list from database

At the moment one has to do this by hand:

• Go to db server (or server with access to db server)
• Run the scripts/members.sql
• scp the csv onto local machine and add to source repo (in data) so others have access to it

Make gather scripts return what they find

In order to conduct meaningful tests it seems to be mandatory to make the gather scripts return their findings...

Beautify (a bit more)

Reconciliation system for users referenced in activity

• Need to fill person_id field on activity.
• Should do this separately from activity creation (in a separate reconciliation process)

Proposed setup:

• Enter: author name to username matches into reconcilication table in gdocs
• Go through reconciling and add to activity table and new column called person_username that links to the person table username column

Questions:

• What happens if no person with that username (need to create a person)
• Dealing with anonymous stuff (leave username blank? create anonymous person)

Rebase Dashboard as front-end atop the Activity API

Suggest:

• Boot this front-end as a Brunch project. This handles vendor libs, dependancies, minification, Backbone model/view/controller seperation, test suites, templating, and intermediate languages of your choice such as...
• CoffeeScript! Porting is painless. Smile as the code shrinks, closures and object bindings are expressed purely in syntax, and global scope is never again polluted.
• Connect to the Activity API instance for nearly-real-time data collected by scrapers running in the cloud. (Thanks Heroku).
• Get a minimal viable app which generates a map of users via Recline. ("Back to where we started"...?)

Add OKFestival 2013

Google analytics for OKFestival 2013 will be set up in the next few days, courtesy of Theo.

So far:
Website http://okfestival.org/ (unless we go online with http://okfestival.org/2013 – tbc atm)
Facebook http://www.facebook.com/OpenKnowledgeFestival
Twitter https://twitter.com/OKFestival

Thank you!

Add some unit tests

E.g. for:

• members.csv processing
• webstore connection

Track a comparison graph of working group mailing lists

Import new logo graphics

The official OKFN logo is now a more saturated teal colour; overwrite the dashboard embedded graphics.

Add PyBossa

Add PyBossa projects https://github.com/PyBossa specifically:

• https://github.com/PyBossa/pybossa
• https://github.com/PyBossa/pybossa.js
• https://github.com/PyBossa/pybossa-client

Web Usage Stats in Dashboard

Google Analytics ranking of all our sites in the blog farm on the basis of unique visitors

okfn-labs mailing list not listed on dashboard for labs

See http://okfnlabs.org/dashboard/#project/labs

BTW: would love to fix this myself if there were instructions (have i missed them?) :-)

Remove datahub-news from the mailing lists for ckan

Announce Post

Convert geonames data to a geojson object

Rather than storing geonames data in members.py store a geojson object (see http://geojson.org/geojson-spec.html). Suggest:

{
  type: "Feature",
  geometry:  {"type": "Point", "coordinates": [geonames.lng,  geonames.lat]},
  properties: {
    label: geoname-name,
    geonameId: geonames.geonameId
  } 
}

Backbone model for Activity

Track Facebook stats

Track Facebook stats in same way as you track Twitter with nice little box

http://www.facebook.com/OKFNetwork

Try out DataBrewery for ETL process

Things that would be needed:

• ds (data store) for ElasticSearch
• Conversion of existing code

Graphs seem reverse sorted and data is out of date

See e.g. http://okfnlabs.org/dashboard/#project/recline

This lists watchers as 545 but number is 631. Similarly graphs seem to go backwards in time.

Add source_id attribute to activity

When creating activity set service field to be service identifier (section name in dashboard.cfg)

Add Working Groups

Add main Working Groups (similar to Projects I imagine)

Project Overview Information

See US#4 and US#3. Suggest

• Some measure of attention/reach (can we summarize into 1 or 2 key values)?
• Activity summary: can we summarize activity into one value - we could then show in a sparkline or similar. Alternatively we could have a simple colour or symbol display (e.g. Green for “increasing/high” red for “decreasing/low”) to convey:
  • Is activity high or low (relative to what?)
  • Is activity increasing of decreasing (month on month)
  • Date of most recent activity

Create Colander schema for Activity and use it on scraped data

Use schema to validate and transform incoming data.

Check out Crowdbooster

We've got some historic snapshots of interesting social networking data in Crowdbooster.

There's no API, so we can't feed off this source directly. But we could pull out the historic data and track these metrics (in particular, Facebook) ourselves.

I'm just saying: Before writing this all from scratch, check out how Crowdbooster have done it to minimise reinvention. Community team just want a couple of key metrics in one place, and Crowdbooster is not a very Open place to have all that.

Main mailing lists to highlight

• OKFN Announce - our main news and fundraising site linked to on the main site
• OKFN Discuss - general discussion about OKFN issues

Not sure of the value of having OKFN Help or OKFN Coord there...

Switch to new Domain Model

Domain Model is documented on http://wiki.okfn.org/Community_Dashboard. Main changes:

• Services -> Sources and in gdocs (plus different structure)
• Project object reffed from activity

Cron jobs to pull github info seem to be failing

See e.g. http://okfnlabs.org/dashboard/#project/ckan which does not have November data ...

Compute stats and cache in stats table

Hard to do until webstore gives us totals / counts - see okfn/webstore#29

Tests for JS code

List main sources and put in sources table

Add them to Sources gdoc. Process:

• For each project in projects list
• Get list of sources of activity e.g. blog posts, code commits, wiki changes etc.
• For each of these add an entry to the sources spreadsheet following the structure suggested there

Switch from Webstore to DataStore (ES)

Backbone models for Person

List and update project / topic listing

Add to and update projects / topic list gdoc

• Enter directly into spreadsheet or (better) via http://labs.okfn.org/spreadsheet/
  • May need to generate images (separate task -- instructions on that form)

Once complete we should move to issue #2 (add sources for projects / topics)