genenetwork / genenetwork2

GeneNetwork (2nd generation)

GeneNetwork version 1

Please see https://github.com/genenetwork for the most recent developments on GeneNetwork on GitHub.

Service initiated January, 1994 as The Portable Dictionary of the Mouse Genome and June 15, 2001 as WebQTL.

Funded by the National Institutes of Health and University of Tennessee Center for Integrative and Translational Genomics

For background see: http://en.wikipedia.org/wiki/Genenetwork

===========

Database

Genotypes

A number of directories contain data that are not checked into git. See this.

Production

For production

gn1@tux01:~/production/gnshare/gn$ . .guix-deploy

Note the extra dot. Copy and paste the URL in the container:

httpd -f /gnu/store/2g36jjrkn39d0npfrrl1rd7z0s8dp52s-httpd.conf

Zach

zas1024@tux01:~/gn1-zach/gnshare/gn$ . .guix-deploy

and paste

httpd -f /export/local/home/zas1024/gn1-zach/gnshare/gn/etc/httpd-zas1024.conf

Other

Running Genenetwork1 in modern environments can be difficult since its dependencies are no longer actively supported. With guix, however, we can reproduce the environment in which it ran. This is possible since some of the old (deprecated) packages are maintained in guix-past and guix-bioinformatics.

To host genenetwork1 in a guix container, run something like

#!/bin/sh

# Modify paths accordingly; or if you have your channels configured
# appropriately, you don't need to include the "-L" flags

/path/to/pre-inst-env guix environment \
    -L /path/to/guix-past/modules/ \
    -L /path/to/guix-bioinformatics/ \
    --pure \
    genenetwork1 \
    --ad-hoc \
    httpd-with-mod-python \
    [email protected] \
    python24-htmlgen-GN1 \
    python24-json-GN1 \
    python24-mysqlclient \
    python24-numarray \
    python24-piddle \
    python24-pp-GN1 \
    python24-pyx \
    python24-pyxlwriter \
    python24-qtlreaper \
    python24-rpy2 \
    python24-svg-GN1 \
    -- \
    httpd \
    -f \
    $(/path/to/pre-inst-env guix build -L /path/to/guix-past/modules/ -L /path/to/guix-bioinformatics/ -e '(@ (gn services gn1-httpd-config) GN1-httpd-config)') \
    -T

This runs genenetwork1 on port 8042. To change the default port, modify: GN1-httpd-config in gn1-httpd-config.scm. Other default params you can modify are:

• pid-file (default: "/tmp/guix-gn1/httpd-genenetwork1.pid")
• error-log (default: "/tmp/guix-gn1/httpd-genenetwork1-error-log")

Ensure the above files exist and your user can write to those directories(i.e. /tmp/guix-gn1).

LICENSE

Released under Affero General Public License 3 (AGPLv3)

genenetwork2's Issues

Feature that warns the user of outliers before mapping is missing from GN2

GN1 has a feature that pops up something warning the user if any outliers exist in the trait's sample/strain data and I believe it will automatically remove them as well. At the very least, we need to add back in the warning.

While this feature exists in GN1, the pop-up looks a little strange (aesthetically) so it might be a better idea to not borrow all the code.

Change the genotype textual header to be JSONized

From @pjotrp on May 5, 2015 22:4

Copied from original issue: zsloan/genenetwork2#15

Look for unsafe SQL queries

Before going live we should go through SQL queries that get modified from forms - they may appear as security breaches.

Need to fix script that updates search page drop-downs

The script that generates the drop-down fields on the main page needs to be updated so that it checks if there is a phenotype data set for that group instead of just automatically adding "(group name) Published Phenotypes" to the drop-down.

I also need to double check that phenotype data really doesn't exist for these groups/data sets. If it does exist, the PublishFreeze table in the database needs to be changed accordingly.

0.50-gn2-pre1

From @pjotrp on April 2, 2015 7:28

Copied from original issue: zsloan/genenetwork2/pull/3

Small version GN database (<=2GB)

Aliases/location of genotypes are not shown on trait page

Example: http://gn2.genenetwork.org/show_trait?trait_id=D3Mit316&dataset=B6BTBRF2Geno

Wildcard (*) search doesn't work

See https://dba.stackexchange.com/questions/19639/why-does-match-fulltextsearch-against-in-boolean-mode-return-nothing

Search results Download button doesn't work

From @zsloan on May 5, 2015 18:8

Nothing happens when the Download button is clicked; I don't think this was ever implemented.

Copied from original issue: zsloan/genenetwork2#9

Graceful handling of errors

When we hit an error the webserver should give a friendly message.

Need Partial Correlation feature

This is probably medium priority; it's higher priority than something like Compare Correlates, but probably isn't necessary to have before release.

Merge textual representation of genotype/phenotype

From @pjotrp on May 5, 2015 21:15

At this point we have two textual representations of these formats. They need to be merged into one.

Copied from original issue: zsloan/genenetwork2#12

Need to remove all uses of cursor for database connection

From @zsloan on March 26, 2015 21:43

Instead we should use the flask g import and the syntax "g.db.execute(the query)"

Copied from original issue: zsloan#2

Exporting trait data doesn't work

From @lomereiter on May 7, 2015 13:40

Neither to Excel nor to CSV.

Copied from original issue: zsloan/genenetwork2#21

Traits are assumed to be log-scaled

From @lomereiter on May 6, 2015 12:10

E.g. page for the trait with id 10001 (where the unit is mg/dl) shows inadequate range statistics

Copied from original issue: zsloan/genenetwork2#19

First CUDA implementation for pylmm

CUDA matrix multiplication and transpose are low hanging fruit for pylmm.

pylmm permutations not working

Whatever you fill in the field it runs pylmm once.

Search terms that apply to multiple data set types only work for one data set type

The relevant file for this issue is wqflask/wqflask/do_search.py

For example, the LRS search (for example "LRS=(23 46)" or "LRS>30") should be usable for both phenotype and mRNA assay data sets.

Currently, each search term's function inherits from a parent class. Right now LrsSearch inherits from MrnaAssaySearch, which means that it only works for mRNA assay data sets. Ideally there would be some way to choose which class to inherit from based on what the data set type is, but I'm not sure how to do this.

Create smooth installation for GN2

No control markers found for some traits + results in field containing just a comma

Here is an example trait where this is the case: http://gn2_zach.genenetwork.org/show_trait?trait_id=236263_at&dataset=GSE5281_F_RMA_N_0709

I'm not sure if it makes sense for it to not be finding any control markers in the first place, but even if it does it shouldn't just show a comma.

Change the genotype textual header to be JSONized

From @pjotrp on May 5, 2015 22:4

Copied from original issue: zsloan/genenetwork2#15

Need to add external links to trait page (RNA Seq, Verify from GN1)

From @zsloan on May 5, 2015 15:16

Copied from original issue: zsloan/genenetwork2#6

Add pylmm gwas (Hs)

Combined search doesn't work for most search types

From @lomereiter on April 22, 2015 17:56

Only a single class, MrnaAssaySearch, has run_combined method, all the other classes in do_search.py are required to have this method since this commit but don't have it.

Copied from original issue: zsloan/genenetwork2#5

Global search

Basic Statistics figures need to update with changed values on the trait page

In GN1 the user has to hit the "Update Figures" button under Basic Statistics to update them based upon changed values in the "Review and Edit Data" part of the page, but since the new figures are created with javascript they should be able to update dynamically.

Handle errors of pylmm gracefully

RIF Search seems to return the wrong (different from GN1 results and is slow)

A search for "RIF=mitochondrial" in the "Hippocampus Consortium M430v2 (Jun06) PDNN" data set returns over 7000 results in GN2 while it only returns over 3000 in GN1, and it also seems to take 2-3 times as long to run.

While I wrote the original code, I think it might be best for Lei to take a look at this since he is better with writing queries.

Error on search result page for some phenotype datasets

From @zsloan on May 5, 2015 17:10

This occurs mostly for human and other non-mouse species phenotype data sets. I think it has something to do with those groups not showing up in the InbredSet table in the database, though I've also seen the issue with a couple of the groups that do seem to be there.

Copied from original issue: zsloan/genenetwork2#7

Interval Mapping needs to be removed for human traits (and others without .geno genotype files)

From @zsloan on May 5, 2015 22:40

It may be possible to edit the interval map code to accept other genotype files, but it probably isn't necessary seeing as we have other mapping methods that are superior to qtlreaper's (though maybe not as fast?).

Estimated time/difficulty: Easy

Copied from original issue: zsloan/genenetwork2#16

Need to update tutorial for GN2

There is no tutorial page for GN2 like the one that exists for GN1, so we should probably create one at some point.

Large files need to be stored in shared directory

From @pjotrp on May 5, 2015 21:18

The idea is to have a shared storage for a server. E.g.

STORAGE=/export/local/gn2/storage

This can be set in the configuration for every running server. Within the storage dir we'll use MD5 values (of the contained files) to uniquely set the dirname, e.g.

/export/local/gn2/storage/d849b0238cd26434237b28498628837d/files...

That hash value will be used to locate the data.

Copied from original issue: zsloan/genenetwork2#13

Error when doing correlation with human mRNA assay data sets

An error is returned when the correlation is calculated for seemingly any human mRNA assay data set, for example: http://gn2_zach.genenetwork.org/show_trait?trait_id=210412_at&dataset=GSE5281_F_RMA0709

This seems to be related to the part that calculates the tissue correlation, as the error is in an SQL query that looks at the TissueProbeSet tables.

Install Arvados as a test

Arvados is FOSS and we can install it on a single server (it needs a
VM environment):

https://arvados.org/

It can run on docker (I think), so best to try on an independent
machine or even your laptop. Alternatively install in the Cloud.

To try Arvados you can get an account with

https://curoverse.com/

We just want to get a little idea of deployment (issues). They have
documentation on the websites.

Basic statistics for datasets with case attributes are not shown properly

E.g. http://gn2.genenetwork.org/show_trait?trait_id=210749_x_at&dataset=GSE5281_F_RMA0709 - no stats, histogram and bar chart.

Histogram bin size is too small

There shouldn't be gaps on the histogram like this (24 bins for 99 samples is way too much):
http://gn2.genenetwork.org/show_trait?trait_id=1436869_at&dataset=HC_M2_0606_P

Need to make all paths relative to the path specified in the config file

From @zsloan on March 18, 2015 16:19

Need to make all paths relative to the path specified in the config file

Copied from original issue: zsloan#1

Move pylmm out of the main source tree

Pass large files through the file system (rather than Redis)

Reduce number of SQL queries while loading trait page

(once I started to use remote DB, excessive queries became very apparent)

Here's an iteration over all tissues, and it should be replaced by a single query, and be done once per dataset group and not per trait:
https://github.com/genenetwork/genenetwork2/blob/53d8a8bff6418a203e3d2565aaeca4fcb5dc0752/wqflask/wqflask/show_trait/show_trait.py#L913-920

UPD: started to work on it - https://github.com/lomereiter/genenetwork2/tree/trait_page_queries

Error when both e-mail and password fields empty in log-in page

From @zsloan on May 5, 2015 17:14

An error page is returned when both fields are left empty; could either use some pop-up after the user hits log-in or could track whether the fields are empty before they hit log-in.

Copied from original issue: zsloan/genenetwork2#8

"Remove trait" missing from collections

From @zsloan on May 5, 2015 21:3

The option to remove traits is missing from the collection page. This option used to exist, so it must have been either mistakenly or purposely removed/commented out for some reason.

Copied from original issue: zsloan/genenetwork2#11

Need to add Network Graph feature

Though it probably isn't necessary to have prior to release, this is one of the more commonly used features in GN1, so it would be good to port it over to GN2.

Add Phenotype All Species search (exists in GN1)

From @zsloan on May 5, 2015 21:49

The All Species search is not yet implemented, but Lei's global search code should be able to take its place once it is finished.

Copied from original issue: zsloan/genenetwork2#14

Consistency regarding JS libraries

From @lomereiter on May 6, 2015 10:26

jQuery and D3.js are fetched from external websites, while most other libraries are stored locally

Estimated time/difficulty: Easy

Copied from original issue: zsloan/genenetwork2#17

Database connector timeout

Need to replicate the issue for testing/fixing

Samples not separated by group on Trait page

From @zsloan on May 5, 2015 18:33

For example, the BXD strains are not separated from the MDP strains in the strain values portion of the page. I feel like this used to work at some point, though ideally the user would be able to customize which sets of strains he/she wants to use.

Copied from original issue: zsloan/genenetwork2#10

Synchronize with latest pylmm

From @pjotrp on April 19, 2015 10:47

Latest edition of pylmm and tested for integration. Note that human may not work.

Copied from original issue: zsloan/genenetwork2/pull/4

Position Search isn't working

Position search, for example "POSITION=(chr1 25 30)", is not working yet. I think that it used to be implemented and working for mRNA Assay data sets but ran into the same problem as I mentioned for the LRS search with it applied to multiple data set types (in this case both Genotype and mRNA Assay). Since @lomereiter seems to think it's okay to create a separate subclass for each data set type, I'll try and implement it that way.

Need to add plink genotype files for other human data sets or remove the mapping methods that use them

Currently the plink .bed/.map/.fam files used by rqtl, GEMMA, and pylmm only exist for a couple data sets, so if you try to run these mapping methods on any other data set it will throw an error (or even worse, not throw an error). At the very least, we shouldn't show these mapping methods for data sets that don't have the necessary plink files. Specifically, we should probably replace the code that just says "if human, show these mapping methods" with something that checks if the relevant genotype files exist.