tripal / tripal_analysis_expression

Extension module for the Tripal toolset to show differential expression data. This module was made for Drupal 7, Tripal 3, and Chado 1.3.

Tripal 4 (Drupal 10)

               _   _                 _____                 _                                  _
     /\       | | (_)               |  __ \               | |                                | |
    /  \   ___| |_ ___   _____      | |  | | _____   _____| | ___  _ __  _ __ ___   ___ _ __ | |_
   / /\ \ / __| __| \ \ / / _ \     | |  | |/ _ \ \ / / _ \ |/ _ \| '_ \| '_ ` _ \ / _ \ '_ \| __|
  / ____ \ (__| |_| |\ V /  __/     | |__| |  __/\ V /  __/ | (_) | |_) | | | | | |  __/ | | | |_
 /_/    \_\___|\__|_| \_/ \___|     |_____/ \___| \_/ \___|_|\___/| .__/|_| |_| |_|\___|_| |_|\__|
                                                                  | |
                                                                  |_|

Compatibility

Tested on +

Drupal	10.2.x	10.3.x
PHP 8.1
PHP 8.2
PHP 8.3

Code Coverage

This project uses Code Climate to determine the quality of our codebase and the coverage of our test suite. Compatibility above is based on passing of this test suite.

Current Timeline

Required for Tripal 4.alpha3 Release

• Display suite / page work
• Importers: Publications
• Search (ElasticSearch + Views)

How to get involved!

This upgrade to Drupal 10 is a community effort. As such, we NEED YOUR HELP!

• To get involved, please join our Tripal Slack and comment in the #core-dev channel.
  • Alternatively, feel free to contact Lacey-Anne Sanderson through Slack direct message.
  • We can use help both with programming, documentation, outreach and welcome all individuals from all backgrounds!
• We prefer automated testing for all Pull Requests (PRs) and are happy to guide you through the process!
  • Just comment on the issue if you need help designing tests
  • If you need help implementing tests see our documentation
• If more documentation would be helpful, please create an issue here and we'll add it.

Documentation

Documentation for Tripal 4 has begun on ReadtheDocs.

• Installation
• Automated Testing
• Developer Guide
• Contribution Guide

Tripal Docker

Tripal Docker is currently focused on Development and Automated Testing. There will be a production focused Tripal Docker soon and if you're interested in helping or providing tips -please join us on our Slack channel!

For more information about how to use our fully functional development docker, see our documentation on ReadtheDocs!

tripal_analysis_expression's Issues

Importer does not provide CV when inserting new property

[site http://default] [TRIPAL ERROR] [TRIPAL_CHADO] chado_insert_property: Cannot find the term described by: Array(    [type_name] => lab host    [cv_name] =>     [value] => University of Copenhagen).
WD tripal_chado: chado_get_property: Cannot find the term described by: Array                                                                      [error]
(
    [type_name] => organism part
    [cv_name] =>
    [value] => Leaf
)

cv_name should be the biomaterial property cv if no CV is selected.

Add units to quantification

We want there to be units to display for expression units. Right now it makes sense to us to put this in quantificationprop.
When creating or editing a protocol of type quantification, it would be great to have a new field pop up for the units.

analysis expression key max value cut off

 var heatMapKey = [{
        'r': 0,
        'g': 0,
        'b': 0,
        'text': 'min expression (' + minHeat + ') -'
    },
        {'r': 0, 'g': 127, 'b': 0, 'text': ''},
        {'r': 0, 'g': 255, 'b': 0, 'text': ' - (' + maxHeat + ') max expression'}];

does not display maxHeat and max expression

biosample importer should allow but not require analyses and guide suggest no analysis

In the case of this module biosamples dont really need analyses. But maybe someone else might want them, so we can make it optional.

Let’s make analysis OPTIONAL for biosample loading. and in my HWG guide, I will specify that we do NOT specify an analysis for loading in the biosample data in general

XML Read Error with Biomaterial Loader

expression map issues on dev site

See http://160.36.205.61:8095/bio_data/1906844

• plot is blank at first. Changing a property selector, or hitting reset, plots.
• legend does not display

permissions need updating

Most permissions are deprecated (adding content types etc).

Need to make sure that we don't need to reimplement some permissions.

Adding new protocol types

Adding new protocol types link is broken admin/tripal/chado/tripal_cv/cv/55/cvterm/add

Deal with zero values on expression plot

Having expression values = 0 offers some awkward looking plots.

Maybe this is OK, because you can remove the nonzero values and the plot looks ok.

before removing

after removing

Administrative page overhaul

The administrator page for this module is sorely out of date and should be updated.

Console error on mouseover of expression map

In the gene_expression field, the below error is sent to console when mousing over a column:

expression.js?oyozky:334 Uncaught ReferenceError: markup is not defined
    at HTMLDivElement.<anonymous> (expression.js?oyozky:334)
    at HTMLDivElement.arguments.length.each.function.n.innerHTML (d3.min.js:3)
    at d3.min.js:3
    at Y (d3.min.js:1)
    at Array.Co.each (d3.min.js:3)
    at Array.Co.html (d3.min.js:3)
    at SVGRectElement.<anonymous> (expression.js?oyozky:331)
    at SVGRectElement.__onmouseover (d3.min.js:1)
```

Manual protocol creation

Text

• Hardware, software, protocol description

• protocol link

• protocol name

Other

• protocol type
• publication

tooltip is off screen

deal with overlapping stuff on the x axis analysis expression

if you have too many things on your x-axis labels overlap

Clicking on biosample linkout is wrong

sends us to biomaterial/ERS1887575 instead of /biodata/entity_id.

Enable user to download expression values

Currently its difficult for a user to download gene expression values to perform their own visualization. We need a button or link to download the raw data as a CSV. The usual matrix format would be best - genes as rows, biomaterials as columns, normalized values filling out the middle. This would be great in 3 places:

1. the analysis record page - download all values
2. the feature page, expression tab - download values for this feature
3. the expression tool - download the values for the features entered into the search box

Allow for long property names

Some property names are super long. Need to either truncate or split into multiple lines/angle X axes

heatmap tooltip malformed for far right column

needs to be tested with more biosamples to see if its a relevant bug. Need to wiggle the mouse a lot to get the tooltip to show up, and when it does, its ugly. Looks fine for the left and middle column.

deal with many biosamples

up to hundreds.

Could make plot scrollable.

Making it downloadable would be very desirable with this change i think.

Make room for legend and title above plot

See attached image

Retrieve all biosample properties not just tissue and treatment

we currently look for tissue and treatment in biomaterialprop under with a call like this:

$tis = chado_get_property([
        'table' => 'biomaterial',
        'id' => $exp->biomaterial_id,
      ], ['type_name' => 'tissue', 'cv_name' => 'biomaterial_property']);

We want to open this up to retrieve all properties associated with a biomaterial (especially from other cv's) and let the user choose which property to color/sort by

heatmap features

Some features are necessary and/or nice to bring the heatmap into line with the individual feature expression viewer.

Critical

• Display all CVterms not just treatment . If that's overwhelming, we can display treatment coming from any CV, not just the biomaterialprop CV.
• Sort by analysis

Nice to have

• Drag to rearrange
• Reorganize or group by property

installationand hook update

additions in hook_update_N MUST BE INCLUDED IN HTE BASE INSTALL.

textwrap legend

set max width to text wrap legend

The 'tripal_analysis_expression features form for heatmap' block appears in every page after module installation

After installing the module, the block 'tripal_analysis_expression features form for heatmap' appears at the bottom of every page. Ideally, it should appear in designated pages or after configured to become active.

At present, we are using neither this form nor the heatmap generated after submitting gene IDs via this form. I had to go via Drupal blocks to inactivate it at four different places to get rid of it appearing in every page of our site.

Biological sample manual input

• Need property input (There is a "tripal" way to do this without creating a new field)
• Contact input needs some description

Migrating function calls

create_biomaterial is defined in tripal_expression_loader but we'll definitely want this packaged with Biomaterals instead of Expression in the upcoming split. For now I've made the biomaterial method private and left the expression loader to prevent dependency issues.

Error checking for uploaded xml file if wrong format

attempting to upload the wrong file (such as expression data) gives an ajax error instead of an incorrect file format error.

Installing causes this error

We should probably report this on tripal_chado if we the our install function needs to run this function.

PDOException: SQLSTATE[42P01]: Undefined table: 7 ERROR:  relation "tripal_cv_defaults" does not exist                                        [error]
LINE 1: DELETE FROM tripal_cv_defaults 
                    ^ in /home/vagrant/code/includes/database/database.inc:2227
Stack trace:
#0 /home/vagrant/code/includes/database/database.inc(2227): PDOStatement->execute(Array)
#1 /home/vagrant/code/includes/database/pgsql/database.inc(111): DatabaseStatementBase->execute(Array, Array)
#2 /home/vagrant/code/includes/database/query.inc(858): DatabaseConnection_pgsql->query('DELETE FROM {tr...', Array, Array)
#3 /home/vagrant/code/sites/all/modules/tripal/tripal_chado/api/modules/tripal_chado.cv.api.inc(1578): DeleteQuery->execute()
#4 /home/vagrant/code/sites/all/modules/tripal_analysis_expression/tripal_biomaterial/tripal_biomaterial.install(42):
tripal_set_default_cv('biomaterialprop', 'type_id', 'biomaterial_pro...')
#5 /home/vagrant/code/includes/module.inc(926): tripal_biomaterial_install()
#6 /home/vagrant/code/includes/module.inc(494): module_invoke('tripal_biomater...', 'install')
#7 /home/vagrant/.composer/vendor/drush/drush/commands/core/drupal/environment_7.inc(143): module_enable(Array)
#8 /home/vagrant/.composer/vendor/drush/drush/commands/pm/pm.drush.inc(1122): drush_module_enable(Array)
#9 /home/vagrant/.composer/vendor/drush/drush/includes/command.inc(361): drush_pm_enable('tripal_analysis...')
#10 /home/vagrant/.composer/vendor/drush/drush/includes/command.inc(212): _drush_invoke_hooks(Array, Array)
#11 /home/vagrant/.composer/vendor/drush/drush/includes/command.inc(180): drush_command('tripal_analysis...')
#12 /home/vagrant/.composer/vendor/drush/drush/lib/Drush/Boot/BaseBoot.php(64): drush_dispatch(Array)
#13 /home/vagrant/.composer/vendor/drush/drush/drush.php(70): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#14 /home/vagrant/.composer/vendor/drush/drush/drush.php(11): drush_main()
#15 {main}

extra mview feature_expression_data

The mview feature_expression_data is present on our dev site, but not found in the code base. It is a near duplicate of the new mview i define expression_feature_all.

• I could use this mview but would need to modify it to include analysis data. But, it isnt created in the install....
• We could delete this old mview in a dbupdate.

Allow and enforce CVTerms for biomaterial property values

Right now we enforce CVterms on the biomaterial prop level, not the corresponding value level.

Ex:
Tissue : leaf

Tissue can be a cvterm, not leaf.

This is a huge issue. Ultimately we want to provide a cvterm doctor linked to a cv browser to implement this.

Allow reordering samples based on treatment

Implementation depends on what is feasible vs flexibility.
Ideally you could drag and drop individual samples however you want, and auto-sort based on any given factor or multiple factors.

Additional features for expression page

• re-implement tool tip
• re-implement key

[] re-implement tool tip
[] re-implement key

Tripal protocols is a DEPRECATED module

We are pretty sure all tripal protocol functionality will be covered by core now.

Let's keep the v2 creation for now but plan on removing once its funcitoning in core.

README.md images

only non-zero values does not function

xml_biomaterial_parser error in release 1.0.2

We are using Tripal2 and have the expression module release 1.0.2 installed. When I load the biomaterial, I get this error when running the drush command:

2017-12-01 13:38:01
Tripal Job Launcher
Running as user 'jhumann'
-------------------
There are 2 jobs queued.
Calling: xml_biomaterial_parser(/main/data/vaccinium/VaccRNAseq-FromMeg_Nov2017/biosample_result_Va.xml, 69, 1008)
call_user_func_array() expects parameter 1 to be a valid callback, function 'xml_biomaterial_parser' not found or invalid function name            [warning]
tripal_core.jobs.api.inc:502
Calling: xml_biomaterial_parser(/main/data/vaccinium/VaccRNAseq-FromMeg_Nov2017/biosample_result_Vc.xml, 13, 1009)
call_user_func_array() expects parameter 1 to be a valid callback, function 'xml_biomaterial_parser' not found or invalid function name            [warning]
tripal_core.jobs.api.inc:502

Any suggestions on how to resolve this would be greatly appreeciated.
Jodi

legend needs rounding

Displays tons of decimals for heatmap, should display none

Y axis label cut off

Adjustment is not quite right.
Also line should be thinner

Implement coloring by property or expression

Allow users to select a property to color bars by.

In addition, expression value should be an option. (even though this is expressed on the Y axis)

Display biosamples organized by analysis

• Group biosamples by analysis they belong to
• Allow user to easily switch between analyses

Download link is direct and could use some visual feedback

#106 implements the downloader class. We had talked about creating an iframe so that the user is notified that their download is being prepared.

Match colors in heatmap vs expression plot

Heatmap goes from blue for 0 gray for mid red for high, let's do this as well.

Expression heatmap does not accept features seperated by newline

Values must be comma seperated ie

FRAEX38873_v2_000001950.1,
FRAEX38873_v2_000001960.1,
FRAEX38873_v2_000001970.1,
FRAEX38873_v2_000001980.1,
FRAEX38873_v2_000002030.1,
FRAEX38873_v2_000001990.1,
FRAEX38873_v2_000002000.1,
FRAEX38873_v2_000002010.1,
FRAEX38873_v2_000002020.1,
FRAEX38873_v2_000001750.1,
FRAEX38873_v2_000001760.1

This is fine except the instructions state:

Feature unique names need to be separated by comma, spaces, tabs or newlines.

Formats that work:

• Comma seperated works

Formats that don't:

• TSV
• Newline
• Spaces

biosample importer bug

Loading the sample data described in the dev dataset results in no properties being added to the biomaterials.

[site http://default] [TRIPAL ERROR] [TRIPAL_CHADO] chado_get_property: Cannot find the term described by: Array(    [type_name] => developmental stage     [cv_name] =>     [value] => SimpleXMLElement Object        (            [@attributes] => Array                (                    [attribute_name] => developmental stage                     [harmonized_name] => developmental stage                    [display_name] => development stage                )            [0] => adult        )).
WD tripal_chado: chado_insert_property: Cannot find the term described by: Array                                                 [error]
(
    [type_name] => developmental stage
    [cv_name] =>
    [value] => SimpleXMLElement Object
        (
            [@attributes] => Array
                (
                    [attribute_name] => developmental stage
                    [harmonized_name] => developmental stage
                    [display_name] => development stage
                )

            [0] => adult
        )

)
.

Load and Display statistics especially p values

The goal is easy- associate p values with a given expression value for a given gene.

The implementation is hard.

What does that p value mean- that it was significantly upregulated? downregulated? Compared to what?
How does it get stored? If its just a p value, then it can be stored similar to units in quantificationprop. If its more than that (ie details of the statistical test... and it probably should be) would this additional info also go in quantificationprop or somewhere else?

verify contact functionality of xml importer

I'm seeing some wonky values in test sites.

Documentation

Documentation needs lots of ❤️ as it still reflects the Tripal 2 module.

Array design input issues

The arraydesign tripal 3 content creation menu has the following issues

selectors

Selectors are text input, and they need to be dropdowns. This is for the following fields:

• manufacturer id (contact)
• platformtype id
• substratetype id
• protocol id

Descriptions

It's not clear what text goes in the array and element dimensions boxes.

Update language in loader to reflect non-microarray data

Use the general description of protocol, acquisition, etc in the analysis_expression loader