guruucsd / ping Goto Github PK

This is a Python framework for accessing and analyzing PING data.

Python 94.23% CSS 0.07% JavaScript 4.78% HTML 0.93%

ping's Introduction

PING automation scripts

This is a set of scripts to use with the Pediatric Imaging, Neurocognition, and Genetics (PING) study. Please see http://pingstudy.ucsd.edu/ for more information about this study.

Installation

All installation dependencies are defined in the requirements.txt file.

Installation steps:

git clone [email protected]:guruucsd/PING.git
cd PING
sudo pip install -r requirements.txt If this fails:
- Make sure you have pip installed: https://pip.pypa.io/en/latest/installing.html
- If it fails installing numpy or scipy, I suggest installing miniconda and installing them via conda install numpy scipy pip
Visit https://ping-dataportal.ucsd.edu/applications/User/requestLogin.php and request access to the PING data portal.

Scripts

The following functions of the data portal are available through this scripting interface:

export.py - Compute some derived measures from the PING data, and export to a local CSV
grouping.py - Plot regressions, separating data by the chosen groupings.
gwas.py - PRELIMINARY--you must request access before using! download GWAS results, or launch a new GWAS study.
scatter.py - Show scatter plots, filtering columns by prefix and choosing computations for x, y, and point size.
similarity.py - Script interface for viewing similarity matrices between different PING measures. Select measures by prefix, and use correlation or partial correlation.
snps.py - Script interface for the SNP browser; find genes associated with SNPs, SNPs associated with genes, and download user data for specific genes. NOTE: PING limits SNP downloads to 5000, so use them wisely!
upload.py - Upload your local CSV with derived measures to the PING data portal, so you can access in the Data browser.

Authenticating in scripts

Scripts must be run with a valid username and password to the PING portal--if you don't have one, you'll need to request one.

There are two ways to pass your username and password to the scripts:

Via the command-line: use parameters --username=[USERNAME] --password=[PASSWORD]
Via the shell: set the following environment variables:
- PING_USERNAME - your PING data portal username (not your email address).
- PING_PASSWORD - your PING data portal password

Example script usage

These examples assume that you've set your username and password via shell environment variables. If you haven't, you'll need to add the --username=[USERNAME] and --password=[PASSWORD] options to the commands below.

python export.py - exports data sheet, including computed measures, to a local CSV file.
python grouping.py MRI_cort_area.ctx Gender - show linear regression for each cortical area measure, with regression by gender overlaid on the same plot.
python scatter.py MRI_cort_area AI:mean AI:std LH_PLUS_RH:mean - show a scatter plot over cortical area measures, of asymmetry index mean vs. standard deviation, with dot size given by the total area (LH+RH)
python similarity.py MRI_cort_area partial-correlation "Left Hemisphere" - show a similarity matrix between all measures with the MRI_cort_area prefix, using partial correlation, for the left hemisphere only.
python snps.py view STK31 - show all SNPs associated with gene STK31, according to the PING genetics DB.

ping's People

Contributors

Stargazers

Watchers

Forkers

bcipolli

ping's Issues

Add continuous integration

When changes are made, scripts can easily break. Having some basic CI via Travis would be great. This requires:

A simple .travis.yml config file
Working tests!

Force a query before downloading PING database to disk

The PING database download is an actual user-specific CSV file that doesn't exist until a user has run at least one query in the data visualization tool.

This code should force such a query (after all, that's easy!) before attempting to download the PING database.

Add doctests

Documentation should have examples right in them!

Add file and function docstrings

Documentation of this code sucks! ;)

Add options for interactive plotting

mpld3 is a python package for plotting (matplotlib) in the browser (`d3). These plots are interactive by default (by allowing zooming) and have a plugin architecture for custom interactivity.

I am creating a dynamic poster for SfN, so mpld3 is ideal.

Add a filter to loop_show_asymmetry

When prepping figures, I found it necessary to filter what keys generated plots, so that I didn't have to draw them all every time.

That was nice. Offer that to users!

Combine correlation & partial correlation plots

Show one in lower triangle, the other in the upper triangle.

Restructure code for analysis using alternate atlas parcellations

The current PING data spreadsheet uses an atlas with 37 parcellations, a bit too rough for many purposes. It would be great to make the code extensible, to easily feed different atlas data to the various analyses.

This will require:

Pushing all functions related to the current PING data spreadsheet into the PINGData class
Extending the PINGData class for new atlases, and overriding those functions.
Making each script retrieve a data object, based on the desired atlas, and reference properties and functions of that data object (and not on PINGData directly).

Once this is done, I should be able to run the current analyses on the Destrieux data that @lcurley shared with me.

Better method for plotting connectivity matrices

Plot correlation and partial correlation on the same figure (lower/upper triangle.
Use this code for anatomical distance estimates.
Perhaps a matrix isn't the right one. Maybe a scatter plot with distance vs. connectivity?

Fix up comparisons with background, behavior

After PCA, prints:

Found 0 TBX keys.
Found 0 FDH keys.

This means that there's no correlation to behavioral measures. Let's get that back in!

Generalize code to PLING, reduce "hard-codedness" of URLs

All of the URLs are copied as full URLs with the "project" field hard-coded. Try:

Creating a URL generation function that accepts a relative path and a project field.

Add progress bar to file downloads

Copy the logic from http://github.com/nilearn/nilearn for file downloads, to get a proper progress bar.

I'm downloading the PING CSV file now, and ... I hope it works, but it's minutes of blank screens right now.

Refactor file download / save logic

It's duplicated in >= 4 places; clean that up!

Add examples

The current documentation lists the scripts, but doesn't provide any examples. When @lcurley took a look at the code, it was hard to figure out where to start!

Best way to add examples is likely through iPython notebooks. Adding to README.md may also help.

GWAS: migrate from measure to prefix

This allows partial matching. In the case of multiple matches, print the matches. only allow launching if unique.

Separate research code from PING portal code

Right now, they're inter-mixed. Make the PING set of tools into a separate package (ping).

Move scripts into ping/scripts/
Make a generic, but simple, framework for computing derived measures.

GWAS: allow output to json, flask

json: dump output into json file.
flask: dump to template, show the manhattan plot.

Enable grouping in more functions

The app has decent filtering functionality right now. However, grouping is not available in most cases.

Filtering: limit the results to a small number of fields.
Grouping: split subjects into groups, and do a full analysis on each group separately (but report together).

Grouping exists in:

export.py / upload.py
grouping.py