m-lab / mlab-vis-client

Measurement Lab Visualizations - Analyzing internet speeds worldwide.

M-Lab Visualizations

This project contains the repository for the Measurement Lab visualizations web client - https://viz.measurementlab.net

mlab-vis-client is a NodeJS application using ReactJS.

General Development Practices

M-Lab follows this general development practice:

• Fork the mlab-vis-client repository.
• Develop in your fork's master or feature branches
• Test and preview changes by deploying to the staging GAE service
• Open a new pull request to merge changes from your fork to upstream master, and request or assign one or more reviewers to perform a code review
• Make changes recommended in the code review
• Merge and deploy the upstream master branch to the production GAE service

Development Environment Setup

• Fork this repo, then clone to your local machine
• Add a git remote for upstream
• Initialize submodules by running git submodule init and git submodule update
• Install NodeJS
• Enter the directory where you've cloned your fork of mlab-vis-client and install dependenciesi: $ npm install

Development Server

We are using webpack's DllPlugin, so we need to build our DLL vendor package before running our standard webpack watch. To do so, run this once:

npm run webpack-dll

This puts all the vendor files in their own bundle so we don't need to scan them when rebuilding our files during development.

If you see the error below, you need to run npm run webpack-dll.

Error: Cannot find module '.../mlab-vis-client/static/dist/vendor-manifest.json'

Now to start the dev server, there are two options:

• Web Server + Webpack Watch in one command: $ npm run dev
• Web Server separate from Webpack Watch
  • Start the web server: $ npm run start-dev
  • Start webpack watch: $ npm run webpack-watch

If you are running against a local API server, you can overwrite the URL that will be used for the api server as part of your run command like so: $ APIROOT=//localhost:8080/ npm run dev. Note that the // and final / are required.

Caching with HardSourceWebpackPlugin

We are using HardSourceWebpackPlugin. If your webpack build is operating strangely, be sure to run:

npm run webpack-clean-cache

This plugin dramatically speeds up build times, but does require you to clean the cache occasionally (when problems arise).

If you see an error in your console similar to:

Uncaught TypeError: __webpack_require__(...) is not a function

Chances are you need to clean the cache. Run the command as described above.

Using webpack-dashboard

If you prefer to have webpack rendered in a dashboard, use two separate terminal windows.

• In one terminal, run webpack with: $ npm run webpack-dashboard
• And in another, run the web server with: $ npm run start-dev

Note that the dashboard adds roughly 500ms to the webpack rebuild time.

Testing

npm test

Building for Production

API_MODE can be sandbox|staging|production (which all are effectively a production build pointing at an API server within the same project) OR development (which is an actual development build.)

API_MODE=staging npm run build

To test whether your build runs as expected you can also run the following command which will also match what should be run by GAE.

API_MODE=staging npm run start

Deploying to Staging and Production GAE Services

M-Lab maintains separate GCP projects for hosting staging and production instances of the mlab-vis-client application.

• staging provides the GAE service, viz
• production provides the GAE service, viz

Deploying to either staging or production requires the Google Cloud SDK to be installed on your development system, and your preferred Google account must be given rights to push to both projects.

Deploying Code to GAE

Deploying assumes that you have already built the files for the respective environment!

To deploy the site to GAE, call:

$ KEY_FILE=<your service key> ./deploy.sh -m sandbox|staging|production

The mlab-vis-client application is configured to deploy to the viz GAE service in the selected project.

mlab-vis-client's Issues

Client ISP Autocompleter

Add in generic `id` field to data objects

Locations should have location key as their id (e.g. nausmacambridge)
Client ISPs should have their asn_number as their id (e.g. AS3215)
Transit ISPs should have their keyed name as their id (e.g. ghanaixp)

This could happen at the API level or at the client transform level. We should use the id when it is available on the objects instead of keying in to their specific properties (e.g. meta.client_asn_number).

API calls should check that all their params are defined before calling

For example, if there is no ISP ID, don't make the API request for ISP data.

Adjust styling of small multiples time series

Currently:

I think it should have the names closer to the axis lines and the headers should be left aligned. Or maybe the names go above the charts. Not sure, but we should play around

Update eslint rules to standardize chart helper function location

I think I prefer to have:

makeVisComponents
setup
update
callbacks
renderHelpers
render

Test count bars should line up with the middle of the line chart points

Right now they are anchored to the beginning of the bars

Maintain colors for ISPs

First pass: keep lines consistent color in location page when adding / removing isps.

Eventual: keep consistent color for ISPS throughout site and between user sessions.

Restore hover behaviour in hour chart

The actions seem to be firing but the data isn't updating. Might have to do with dataByDate being set up by d3.nest

Loading indicators on charts

When charts are loading, they should have some kind of overlay indicating it's happening.

Small multiples line chart

clicking breadcrumbs loses url parameters

Add some sort of a loading indicator to charts

When fetching lots of data, it might take a bit longer for the charts to update. We should have a visual indicator of this for the user so that they don't think things are broken.

Breadcrumbs repeat region/country/continent name

NA / US / Washington / Washington for instance. Should just be NA / US / Washington.

Scatterplot chart

Selected client ISPs can get clipped by the chart

test count y axis labels too dense

Consider giving search API calls their own LRU cache

Right now search calls use the same get and therefore the same LRU cache as all other API calls. This may be problematic in that search involves a ton of get calls as the user enters characters in the search bar, and thus it would quickly destroy the cache.

A solution would be to use a getSearch or similar that used a separate cache for search requests.

Don't repopulate client ISP list when all have been removed

The user removed them for a reason.
it is unkind to add them back.

Validate selected ISP url param

Enforce that we don't get repeat ISP IDs in the string. (e.g., isps=AS7922_AS15169_AS3_AS15169 is invalid)

Line chart tooltip should show date

Currently only shows values, but it would be nice to see the date too. maybe just on top of the x-axis.

line charts need ISP legend

Visualizing child regions when looking at locations bigger than cities

Summary data table

Time range picker

Multi-line chart

• Axis labels
• Legend
• Tooltips on hover

Add in caching for requests

We can use some LRU cache so we don't have to hit the API for recently requested calls. Do this at the request level so we can keep Redux/components unaware of it.

When an error occurs, the UI continuously tries to fetch

This is a side effect of how we handle fetchIfNeeded, since we say yes go fetch if there is an error. Should have some limit on retries before error is a dont-go-refetch problem.

Improve search field labelling

Clear distinction needs to be made between location based searches and ISP based searches.

We also might consider adding location search to the top nav for easy access to location changing functionality.

Test count bars should have a max width so that they are not cropped

Like this:

Selectors get recomputed all the time if they depend on `location`

Since everything is essentially stored in location, the selector is getting recomputed all the time when nothing changes. Try to make them smarter.

Add interaction to small multiple.

Should mirror as much as possible capabilities of main graph.
Should be linked interaction.
similar to: http://projects.flowingdata.com/tut/linked_small_multiples_demo/

Map for location with selected children indicated

adjust y extent to be the same for both download and upload speeds for multi-line charts

Summary line chart

calendar picker should open on the date currently selected or passed via the url

CSV export for multi-series data

If we have more than one line in a time series, exporting the data as CSV becomes more challenging and the current implementation will not suffice.

Show location info and breadcrumbs

Currently we see this:

But we should see the actual bread crumbs and names of places.

Line chart should have a minimum height for the chart area most likely

Or perhaps a set height. Right now the height it is given is split up to fit the legend, which can get very tall if there is a ton of ISPs selected. I dont think the legend getting taller should make the chart shorter.

Similar cities

Hourly chart

• Loading status should be individual to each chart
• Charts should have headers
• Axis labels
• X hover behavior should highlight count chart
• Should have interactive behavior for within-hour
• Should use standardized colors

Check how things are looking on diff browsers

Georgia mentioned there might be an issue, but she isn't 100% sure what it is. We should check if things are rendering ok on Firefox

time picker should keep dates when moving up or down the location heirarchy

Question: Height of small multiple

Should small multiple component be provided a height and constrain the rows of charts to fit within that height (as other components are implemented)?

or should the component grow and shrink given the number of clientISPs that are being displayed (as we originally envisioned)?

Current implementation follows the former option.

Small multiples distribution chart

Exporting line chart as PNG loses font in legend

It should probably back up to a sans-serif instead of serif, which it currently seems to

Test count bar chart

Exclude client isps that are already selected from client isp selection tool

For example, if comcast is already selected, don't show it in the drop down menu.

/location/ gets added to the url many times

When reducing time range, then hovering over legend, the line extends beyond the y axis

Setup Redux store for location page

Proposed store design

locationPage:
  locationId
  viewMetric (download, upload, RTT, retransmission rate)

  selectedClientIsps
    - clientIspId1
    - clientIspId2

  selectedLocations
    - locationId1
    - locationId2

  selectedTime:
    startDate
    endDate
    timeAggregation (day, month, year)

  chartOptions:
    showBaselines
    showRegionalValues

  fixed:
    timeFrame (lastWeek, lastMonth, lastYear)
    xMetric (download, upload, RTT, retransmission rate)
    yMetric (download, upload, RTT, retransmission rate)


locations:
  locationId:
    locationId
    name
    parentLocations

    time:
      startDate
      endDate
      timeAggregation
      timeSeries
      hourly

    fixed:
      lastWeek
      lastMonth
      lastYear
      distribution


clientIsps:
  clientIspId:
    clientIspId
    name

    locationTime:
      locationId
      startDate
      endDate
      timeAggregation
      timeSeries
      hourly

    fixed:
      lastWeek
      lastMonth
      lastYear
      distribution