nlesc / spot

Try the demo

SPOT - extensible facet browser

Preview

 

 

Quick note on privacy

When the data is imported to SPOT, it is not saved in any cloud platforms. Your data stays only in your browser regardless what SPOT instance you are using (SPOT demo instance, SPOT-desktop).

What is SPOT?

SPOT is an open science platform on which researchers can visually analyze and compare the datasets from published scientific journal articles. It is also a visual data analytics tool for multi-dimensional datasets. SPOT allows researchers to recreate representations (plots) in the publications and do more exploration. In this way, static plots in the publications become interactive and researchers can look at the data from different angles. The interactive dashboards for the quick analysis of datasets and easy identification of correlations between variables can be done in only a few clicks. The tool was initially developed for the iDARK project, which aimed to combine and visualize the worldwide data within the most general models of dark matter. The published datasets in this research project are being collected at www.idarksurvey.com. SPOT was designed as a generic tool in order to extend and reuse it in other disciplines than physics. It is quick and responsive, even for data sets containing huge number of records.

More details about SPOT can be found at https://research-software.nl/software/spot

How to run it

Make sure you have git installed on your system.

Prerequisites for running SPOT as stand-alone website using crossfilter:

1. follow the instructions to install node.js:
   • via package manager (suggested)
   • Binaries
2. clone this repository:

   git clone https://github.com/NLeSC/spot.git && cd spot

3. install dependencies:

   npm install

   Note: some dependencies may require node-gyp. If you get errors during compilation of this package, you may need to install following packages on Ubuntu system or equivalent packages for your distribution.

   sudo apt-get install -y build-essential python libpq-dev

4. Create .env file using .env.sample and adjust the settings if necassary
5. start the web the application

   npm start

6. open http://localhost:9966 in a web browser

Building the website is only tested on Linux, but it should work on any OS (Mac OS X for example) that is supported by node and npm.

Hosting the site can be done by any webserver.

Make sure that Javascript is enabled in your web browser. SPOT is fully functional in Google Chrome and Chromium web browsers and it should work in other web browsers. Otherwise, please submit an issue.

Using Docker (tested on Linux)

• Get the latest Docker image

docker pull nlesc/spot:latest

• Alternatively, you can build the Docker image yourself

  docker build --network=host -t nlesc/spot .
  

• Run SPOT in a container

docker run --rm --net=host -ti -p "9966:9966" nlesc/spot

• Open http://localhost:9966 in a web browser

If you want to run it using custom settings in .env file:

docker run --rm --net=host -ti -v $(pwd)/.env:/app/.env  -p "80:80" spot

SQL Database

Spot can also work with a PostgreSQL database, but this requires either a local or a remote service to run. Communication between the client and the database server is achieved by using web socket.

In order to use SPOT with a PostreSQL server, you need to clone the spot-server repository and follow the instructions in the README. In general, these are the steps to follow:

1. make sure that PostreSQL service is runnning.

2. upload your data to the database with the spot-import.js script

3. run the SPOT-server with the spot-server.js script

Desktop version

Desktop version of SPOT is still under development. Available downloads can be found here.

Documentation

SPOT API documentation can be found here.

Credits

Jisk Attema, Faruk Diblen the Netherlands eScience Center

spot's Issues

drag & drop improvement

Drag and drop needs to be improved.

http://interactjs.io looks like a good option.

add opacity setting to bubble plot

Statistics panel

Create a widget showing statistics:

• dropped points per filter
• comparison of selection vs. the rest (A/B)
  etc.

Installation for our Partners at DSM

Message from DSM:
I tried to install but I do not understand where I get the package.
I installed node.js, and in Windows PowerShell tried to run: npm install http://nlesc.github.io/spot
But that does not work, the site does not mention how to get or where to find the package, I think.

chart icons does not appear when using Firefox

Chart icons on Analyze page does not work when using Firefox.
No issues with Chrome.

Docker implementation not working on Macbooks (database)

Because Docker in Mac OS runs in a Linux VM, you cannot connect to localhost. For example, I need to connect to http://192.168.99.100:9966/ to go to the spot tool.

Database connection does not work because localhost is hardcoded:

While this should be (in my current case 192.168.99.100)

Loading bar / icon / indicator on upload

When uploading a data set, there is no visual indicator something is happening in the background. I found myself questioning if the upload was successful or not when uploading a > 50 MB file. I would have preferred the system to give me a loading bar / icon / indicator at the location where the uploaded data set can be activated (as temporary placeholder).

Default range gives "Input is not a number!"

Create plot
Add variable
Click on the variable to edit the range
If the range is very small, it gives the error

npm start gives error

. (master) ~/Code/spot $ npm start

> [email protected] start /home/jiska/Code/spot
> npm run fonts && npm run templates && npm run css-watch & budo -v -P --live --dir public --css app.css client/app.js -- -v

[0000] info  Server running at http://145.100.116.2:9966/ (connect)
[0000] info  LiveReload running on 35729

> [email protected] fonts /home/jiska/Code/spot
> mkdir -p public/fonts && cp node_modules/material-design-icons/iconfont/* public/fonts


> [email protected] templates /home/jiska/Code/spot
> templatizer -d client/templates -o client/templates.js


> [email protected] css-watch /home/jiska/Code/spot
> stylus -w -o public --include-css client/stylesheets/app.styl

events.js:160
      throw er; // Unhandled 'error' event

category count not correct

The count per category is not correct; it is not reset to 0 so it keeps increasing when interacting with the plots.

snackbar

Snackbar would be useful for informing user about progress.
See:
https://getmdl.io/components/index.html#snackbar-section

This can also be used for #6.

Network Graph Issue

Network graph takes only two variables, it should contain all three variable

Requested Feature

Exploratory Analysis:

1. Still a network hairball chart is desirable before the following.

Downstream Analysis:

1. To add a summary statistics below the visualized analysis/triples. For example the most frequent Subject, Predicate and Object.

Cannot read property 'isDatetime' of undefined

I think you can reproduce this if you rename a facet when it is in a plot

Uncaught TypeError: Cannot read property 'isDatetime' of undefined
    at initChart (http://nlesc.github.io/spot/app.js:4:27283)
    at update (http://nlesc.github.io/spot/app.js:4:29075)
    at child.update (http://nlesc.github.io/spot/app.js:5:244)
    at child.<anonymous> (http://nlesc.github.io/spot/app.js:4:17620)
    at Object.triggerEvents (http://nlesc.github.io/spot/app.js:8:14517)
    at child.trigger (http://nlesc.github.io/spot/app.js:8:13138)
    at child.filter.getData (http://nlesc.github.io/spot/app.js:5:19090)
    at http://nlesc.github.io/spot/app.js:5:19495
    at Array.forEach (native)
    at child.Collection.(anonymous function) (http://nlesc.github.io/spot/app.js:8:2713)

3D plot type is needed

In the line chart, make unselected grey

In the line chart, I can't see which data I selected (before I press 'zoom in'), because the color of the data points (and graph underneath the points) doesn't change.

Use font awesome

time-axis for line-chart

time-axis for line-chart is not working, see attached figure

Explicit 'back' button

I would like to have a 'back' button in each screen that lets me go back to the previous screen (and gives me confidence that what I just did is saved).

view only mode

View only mode in which a user can see the charts would be useful. Analyze function will be disabled in this mode.

download session broken in firefox

Downloading a session from spot is broken in firefox. Works in chromium.

If you refresh http://nlesc.github.io/spot you go to a different site

new features to add

• zoom functionality for charts
• progress bar
• better mobile device support

Network Graph Issue

Unconnected nodes should not be displayed.

Unit testing fails for factory_spec.js

Testing doesn't work with Plotly. The file was renamed as factory_spec_FIX.js so that Jasmine will ignore it.

back-end for SPARQL is needed for ODEX4all project

In order to connect to SPARQL endpoint (RDF server), a back-end must be implemented.

The JS library below that can be used:
https://github.com/RubenVerborgh/SPARQL.js

node-Postgres bridge

This can be very handy to implement data uploading/downloading.
https://github.com/brianc/node-pg-copy-streams

Not all variables are shown in small resolution

If the window is not wide enough to show all variables in the 'analyze' tab, there is no way to scroll to the right. Either this should be possible, or the variables should be placed in a new row.

Network Graph issue

Network should be updated with the data selection takes place

missing files in fonts folder

Some files are missing in fonts folder:

/fonts/MaterialIcons-Regular.woff2
/fonts/MaterialIcons-Regular.woff
/fonts/MaterialIcons-Regular.ttf

Graphs or network plots

Add a widget showing a network.
Possible libraries:

• http://sigmajs.org/
• http://visjs.org/index.html

Testing SPOT for visualization and analysis of ODEX4all triples

I am trying to visualize and analyze my data (ODEX4all project) using SPOT. Here I wrote down some of the issues I came across while working with SPOT:

• Is it possible to remove uploaded data files?
• In case of 3 loaded data files, the button to upload data has moved or is gone?
• When changing the column selection; figures are not reset. It is hard to drag and drop the column number to remove the previously selected column and add a new column number.
• When label(s) is (are) removed from a graph, is it possible to get it (them) back without having to start all over again?
• Generation of network from 3 columns does not seem to work
  I would be happy to learn more about the possibilities of SPOT. If you like we could meet at NLeSC to discuss the use of SPOT within the ODEX4all project ([email protected]).

Add error bar option to relevant chart types

mobile device support is needed

found these issues so far:

• resolution is not adjusted
• drag and drop does not work
• icons and images are too big (due to scaling problem?)

zoom functionality is needed

Question: how to recreate a plot from idarksurvey with spot?

Not sure if this is the right place as it technically is not an error. I tried using the green points plot on idarksurvey.com (data_pf.csv that I sent a while back) to recreate the default scatter plot of idarksurvey.com. This is a scatter plot of the columns nmass vs m1.

I upload the csv file and select a bubbleplot chart. When I click on the view mode instead of the edit mode I get one huge bubble with a few bubbles on it. I can click on the zoom or fullscreen button but nothing happens. The plot is not very insightful compared to the demo one. See the two screenshots:

My question is: how can I create the idarksurvey plot with SPOT? I understand that clustering is needed, but the amount of clusters should be user definable (maybe even turned off, so n_clusters = n_points, with a warning if there are too many clusters in that case).

categories limited

At the moment, due to UI restrictions, the total number of categories is limited to 50 in the SQL backend.
The crossfilter backend is not limited.

Binning settings are reset when zooming in

If you zoom in on specific data, the settings about how to bin the data are lost.

Indicate number of required facets

When constructing a new graph, it is not clear how many facets are needed as input. Pure intuitively it can be understood that a histogram needs at least one, but i did not have this intuiting for e.g. the network chart.

Implement separate colors for each pie piece of chart

Not from me but Anneke :-) !

Button to select all data

When I have zoomed in on a particular slice of the data, it seems like there is no way to go back to all the data again. I would like to have one reset-selection button to select all data again.

Axis labelling

Is it possible to put the labels on the axis as well as on the title?

Icons on toggle button in top right of Analyze

Currently the toggle button in the top right plays a vital role in the workings of the Analyze tab, but it is not clear that it does so: it is not immediately clear what happens when you would switch it. Addition of two icons on both its left and right side would solve this problem.

remove facet

tick box for each facet and remove button to delete selected facets

Clustering

Implement clustering:

1. as a widget. Clustering can be done as a first attempt on the active facets, using euclidean distance on the bin-numbers, with the bin-count as weighing factor.
2. as an extra dimension. How this works for postgres needs some thought.

Automatically read column names for csv

I would like to have the possibility to indicate that the first row of my csv file is a header, and that the column names are automatically used.

Selecting multiple items is not intuitive

Selecting multiple data points (for example in the bar chart or pie chart) is not always intuitive. I would expect that it selects each bar/part/point that I click individually, and unselects it again when I click it again.

File upload progress

It is not clear if the data file is uploaded successfully. A progress bar can be added to home page.

templates.js needs to be generated before npm start

npm start supposed run npm run templates but it looks like it does not work.
npm run templates must be run before.

create test/testing branch

a new branch called 'test' or 'testing' would be useful to try new and unstable features.