tripal / tripal_elasticsearch

Tripal 4 (Drupal 10)

               _   _                 _____                 _                                  _
     /\       | | (_)               |  __ \               | |                                | |
    /  \   ___| |_ ___   _____      | |  | | _____   _____| | ___  _ __  _ __ ___   ___ _ __ | |_
   / /\ \ / __| __| \ \ / / _ \     | |  | |/ _ \ \ / / _ \ |/ _ \| '_ \| '_ ` _ \ / _ \ '_ \| __|
  / ____ \ (__| |_| |\ V /  __/     | |__| |  __/\ V /  __/ | (_) | |_) | | | | | |  __/ | | | |_
 /_/    \_\___|\__|_| \_/ \___|     |_____/ \___| \_/ \___|_|\___/| .__/|_| |_| |_|\___|_| |_|\__|
                                                                  | |
                                                                  |_|

Compatibility

Tested on +

Drupal	10.2.x	10.3.x
PHP 8.1
PHP 8.2
PHP 8.3

Code Coverage

This project uses Code Climate to determine the quality of our codebase and the coverage of our test suite. Compatibility above is based on passing of this test suite.

Current Timeline

Required for Tripal 4.alpha3 Release

• Display suite / page work
• Importers: Publications
• Search (ElasticSearch + Views)

How to get involved!

This upgrade to Drupal 10 is a community effort. As such, we NEED YOUR HELP!

• To get involved, please join our Tripal Slack and comment in the #core-dev channel.
  • Alternatively, feel free to contact Lacey-Anne Sanderson through Slack direct message.
  • We can use help both with programming, documentation, outreach and welcome all individuals from all backgrounds!
• We prefer automated testing for all Pull Requests (PRs) and are happy to guide you through the process!
  • Just comment on the issue if you need help designing tests
  • If you need help implementing tests see our documentation
• If more documentation would be helpful, please create an issue here and we'll add it.

Documentation

Documentation for Tripal 4 has begun on ReadtheDocs.

• Installation
• Automated Testing
• Developer Guide
• Contribution Guide

Tripal Docker

Tripal Docker is currently focused on Development and Automated Testing. There will be a production focused Tripal Docker soon and if you're interested in helping or providing tips -please join us on our Slack channel!

For more information about how to use our fully functional development docker, see our documentation on ReadtheDocs!

tripal_elasticsearch's Issues

server deleting is not working

can't edit or delete a remote server

Limit the categories to what you have in the results when creating a collection

Add template files

We need template files for site wide search and gene search

admin menu index

see for example tripial content types.

All tab headings are an entry in a landing index.

Update failure

drush updatedb --yes
Tripal_elasticsearch  7212  @throws \Exception
Do you wish to run all pending updates? (y/n): y
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [gene_search_index] as the final mapping would have more than 1 type: [gene_search_index,                         [error]
chado.feature]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [gene_search_index] as the final mapping would have more than 1 type: [gene_search_index, chado.feature]"},"status":400}
ERROR: failed to add new elasticsearch fields (related_features) due to the following error
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [gene_search_index] as the final mapping would have more than 1 type: [gene_search_index, chado.feature]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [gene_search_index] as the final mapping would have more than 1 type: [gene_search_index, chado.feature]"},"status":400}Performed update: tripal_elasticsearch_update_7212                                                                                                                                                                   [ok]
'all' cache was cleared.                                                                                                                                                                                             [success]

ES warning: string field deprecated

looking at the output from my ES image:

elasticsearch_1  | [2018-01-31T18:43:33,868][WARN ][o.e.d.i.m.StringFieldMapper$TypeParser] The [string] field is deprecated, please use [text] or [keyword] instead on [raw]

do we care about this? Should we update our string types to text/keyword?

child/parent features need to be searchable

features are linked in chado via feature_relationship. We are adding SSRs to genes via feature_relationship, and moving our entities from mrna to GENE for chestnut, where none of the sequences are even loaded into the gene. see:

• statonlab/hardwoods_site#226
• statonlab/hardwoods_site#35

At the least, child/parent feature NAMES need to be indexed by ES.

Error sorting Gene Search

[Mon Sep 10 12:03:21.769619 2018] [:error] [client 66.249.64.89:45390] [TRIPAL ERROR] [TRIPAL_ELASTICSEARCH] Feature search in organism Quercus                  rubra has failed with the following                   message. {"error":{"root_cause":[{"type":"parsing_exception","reason":"request does not support [sort]","line":1,"col":200}],"type":"parsing_exception","reason":"request does not support [sort]","line":1,"col":200},"status":400}

index management form fails if no ES servers

index management page is blank if no elasticsearch server is connected at /admin/tripal/extension/tripal_elasticsearch/indices_management.

Confusing if you're setting it up for the first time.

Cross site search see-all link to remote sites that have appended root path don't work

Remote sites like Treegenesdb.org/Drupal don't correctly parse the View All link in the cross-site search page.

Try clicking on the view all button for a remote site on this page to see the effect: https://hardwoodgenomics.org/cross-site-search?terms=*&category=null

Error in Gene Search

Notice: Undefined index: organism in tripal_elasticsearch_gene_search_index_results() (line 259 of /data/www/html/hardwoods_dev/drupal-7.52/sites/all/modules/custom/tripal_elasticsearch/includes/tripal_elasticsearch.gene_search.form.inc).

Error when deleting nodes

Notice: Undefined property: stdClass::$id in tripal_elasticsearch_node_delete() (line 658 of /data/www/html/hardwoods_dev/drupal-7.52/sites/all/modules/custom/tripal_elasticsearch/tripal_elasticsearch.module).
Warning: strtr(): The second argument is not an array in syslog_watchdog() (line 115 of /data/www/html/hardwoods_dev/drupal-7.52/modules/syslog/syslog.module).

feature search on organism returns results from all species

I picked the example IPR term from F. pennsylvanica and get results from all species

http://160.36.205.61:8095/organism/Fraxinus/pennsylvanica?search_term=IPR020575&ds_pane=Feature%20Search

Feature search from the organism should auto-include a filter organism == current bundle organism.

Node/Entity Update Hooks are Possibly Broken

What happens when an entity or node gets published with status=0 and then updated to status=1?

Allow users to update indices

See statonlab/hardwoods_site#169

Honor the expose option

Currently, whether the expose index option is checked or not, the index is always available for cross site search.

Cross site search returns wrong categories

Cross site search returns the name of the bundle rather than the label.

Add collections to Organism's Feature Search

statonlab/hardwoods_site#178

No permission to download search results as csv file

No permission to download search results as csv file. Need to configure the file system in drupal.

Add category to search results

Add which category to the website/entities search results

Cross site search is broken when specifying categories

documentation needs to link to next/previous section

At top, link previous instruction. at bottom, link next.

Gene type not showing on live site

No idea why. It shows up on my local dev site but on main dev or live.

when no alive nodes, fail gracefully

currently a search fails with this error

Elasticsearch\Common\Exceptions\NoNodesAvailableException: No alive nodes found in your cluster in Elasticsearch\ConnectionPool\StaticNoPingConnectionPool->nextConnection() (line 51 of /data/www/html/hardwoods_dev/drupal-7.52/sites/all/libraries/elasticsearch-php/vendor/elasticsearch/elasticsearch/src/Elasticsearch/ConnectionPool/StaticNoPingConnectionPool.php).
presumably when ES connection is down.

would be better if it provided an error message but didnt break site.

Add documentation for CentOS and Ubuntu PM Installation

Let's add links to docs:

• https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html#deb-repo

collections and bundle types

User is currently asked to select a bundle type when creating a collection. It is choose one only.

1. Collections now supports multiple bundle types per collection.

2. The search interface only lets you filter a single bundle type per search.

I think the correct way to do this is to a) do away with the first question, autodetect what bundles are in the search, and b) make the "filter by bundle" sidebar multiselect.

Migration errors

error with Tripal_elasticsearch 7206 and 7207

Cannot add field tripal_elasticsearch_queues.type: table doesn't exist.                                                         [error]
Performed update: tripal_elasticsearch_update_7206                                                                              [ok]

Node type should be the label

Nodes index uses the machine name to store page type. Instead, it should use the human label.

failing to provide host breaks site

This exception prevents content from loading.

Exception: A host was not provided. Please set an Elasticsearch host through the admin interface.
 in ESInstance->__construct() (line 50 of /Users/chet/UTK/tripal/sites/all/modules/custom/tripal_elasticsearch/includes/Elasticsearch/ESInstance.php).

Is this as intended?

Feature Search in Organism Pages Fails

Feature search field in organism pages does not return results based on the current organism but any organism that matches the search terms. Need to refine the organism index to do a "must" query rather than "should"

Categories block doesn't need pages config

Since the block does not render until there are results, we should remove the pages config since most people don't have those pages until after the module is created which makes the block not render at all.

Progress is Inaccurate

The progress tables are not being updated correctly by the end of each run.

The website search block disappeared!

I can't find it in the blocks page ...

Updating index.max_result_window must be done after creating the index

Elasticseach by default returns upto 10,000 results and errs when requesting anything more than that. The way to update the index is using

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.max_result_window" : "1000000"
}'

Question is can this be done while the cluster is up and running? Maybe we should look into a way that assigns it this setting when creating the index.

Add blast annotation to site wide search

statonlab/hardwoods_site#174

Downloading FASTA from Gene Search as Anonymous User Fails

When downloading a large fasta file as anonymous user from the gene search page, the user gets a weird binary encoded content displayed on the page.

collection problem: too many entities (need to udpate with API?)

• the tripal collections API has changed slightly as of this week.

• I created a collection with twenty two thousand entities. However, the resulting collection has 40k entries?

Use ReadTheDocs

would be nice to have RTD here

Feature Search in Organism Pages is not Working

Try searching for IPR terms or blast results. Nothing comes back!

Error getting property of non-object

PHP Notice: Trying to get property of non-object in /var/www/html/sites/all/modules/custom/tripal_elasticsearch/includes/Jobs/NodesIndexJob.php on line 68 pid 23318
WD tripal_elasticsearch:                                                 [error]
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed
to parse"}],"type":"mapper_parsing_exception","reason":"failed to
parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor
detection can only be called on some xcontent bytes or compressed
xcontent bytes"}},"status":400}
[site http://default] [TRIPAL ERROR] [TRIPAL_ELASTICSEARCH] {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}},"status":400}

This is happening on our live site.

Error when host is not available

Error: Class 'Elasticsearch\ClientBuilder' not found in ESInstance->__construct() (line 60 of /Users/Almsaeed/Work/DevSites/Tripal/sites/all/modules/tripal_elasticsearch/includes/Elasticsearch/ESInstance.php).

Data collections url changed

In the most recent tripal version, data collections URL changed so we need to update our collections modal to redirect to the new URL.

update 7212 prints the message @throws \Exception for its description.

updatedb: Tripal_elasticsearch 7212 @throws \Exception

error spamming publishing with elasticsearch down

I published 500 biosamples on my dev site.

For each one i got an error:

WD tripal_elasticsearch: No alive nodes found in your cluster                                                                   [error]

kind of a bummer. here is my current setup:

so i dont even have ES running locally, let alone an idnex to add the entities to.

Gene URLs are not showing up

Feature search URLs are wrong and redirect to the home page.

Build Cross Site Search for Custom Indices

Fields attach hook

Look into a way to trigger a hook when an entities gets a new/deletes a field

Slow gene search indexing

Can we loop through entities and collect annotations instead? Would this be faster?

Maybe we should use a the feature_dbxref table to collect annotations (could be faster).

Update Gene Index When Feature Entities are Added/Updated/Deleted

Since the Gene index only houses published feature entities, we can easily pull the feature record from chado when a feature entity is created, load its annotations and add it to the index.

Remove Header/Menus From Index

Currently we are indexing the entire page, including header and navigation items. One way to do this is to render nodes manually and invoking all node_view hooks. For entities, we need to invoke field hooks.