cancervariants / therapy-normalization Goto Github PK

Update NCIt ETL methods to work with DynamoDB.

We are currently writing normalization routines for each normalizer, but we should move to a single datastore for queries going forward.

This will reduce the amount of code we need to test and maintain.

This is a big issue, and will require several components. I am adding the EPIC label to designate this as the top-level issue, and we can track the related tasks on the Application Refactor project.

other_identifiers should now include RxNorm and ChemIDPlus.

Our recommendations will require handling of non-drug (e.g. radiation) therapy terms.

Transition from sqlite database to DynamoDB.

We can track the related tasks on the Application Refactor Project.

• ChEMBL ETL methods (#46)
• Wikidata ETL methods (#47)
• DrugBank ETL methods (#48)
• NCIt ETL methods (#49)
• Refactor query.py (#50)
• Refactor CLI (#51)

The normalize endpoint should generate a single, merged concept for search terms.

Non-urgent, but I think it'd be helpful at some point to briefly talk about what/how we should be logging across normalizers and sources and make it uniform.

https://www.nlm.nih.gov/research/umls/rxnorm/index.html

Currently, the dgidb branch uses a revised response format from the master branch, where instead of an array of normalizer responses it returns a JSON object for each normalizer class.

I think it is better to maintain the array, but parameterize what normalizers are returned and optionally allow for the DGIdb response format.

Add additional attributes to source meta:

• rdp_url: Optional[str]
  

• non_commercial: Optional[bool]
  

• share_alike: Optional[bool]
  

• attribution: Optional[bool]
  

Drug approval status by locale is commonly requested information. We should capture this in our term normalization effort.

We're using a lot of raw SQL currently for both entering and pulling data -- for development reasons and for better security, we should alter this to use more of SQLAlchemy's feature base.

Currently we focus only on loading data for normalizers, but for this to be production ready / reusable we need to provide methods to update the data files used for the normalizers.

We currently have one lonely unit test implemented in pytest, but should build that out to cover the ChEMBL normalizer and other use cases.

@korikuzma and @jsstevenson have identified GitHub Actions as a good starting point for CI.

Make database updates a stand-alone task that can be called from a command line interface.

Click is a common and useful CLI tool. Typer is a Click-based CLI tool by the FastAPI creators. I mention the latter due to its closeness to the FastAPI toolset, but I think we should implement the CLI using Click directly due to its maturity and active community.

The first CLI util we'll want to make is to run one or more the ETL ops created in #30. By default users should specify which source they want to update, though we should provide an --all flag that can shortcut to everything. We should build this utility with thought on how we would parallelize this I/O bound process.

We should include metadata on data licenses on a per-normalizer basis:

e.g. CC licenses: https://creativecommons.org/

This thread tracks known and public drug terminologies and normalization services. Initial list here: https://docs.google.com/spreadsheets/d/1bhA4RVt7HT2oIMVWeaXcnxFNcUQ4rEl9mB5cBSK8Kzw/edit#gid=0

It would be great to get NCIt concept codes where possible. For drugs, this will be descendent concepts of Pharmacologic Substance.

Right now, we manually download the normalizer's data file. We need to create methods to download from the source's site.

Currently, the normalizer list in query.py instantiates the normalizers on test runs (even when mocked), which raises file not found errors. We need to do some dependency injection or something of the sort to divert it to the correct pathways.

We should consider how we want to handle queries with whitespace.

The below example shows how the addition of whitespace between hyphens can change the results we get.

No whitespace

Whitespace in between hyphens

Our current filtering strategy to get relevant ChemIDplus records (checking for nomenclature tags) removes some entries that our NCIt data links to. So, we want to be able to opt-in records pointed to by NCIt. We need some combo of the following:

• NCIt should be able to tell us which ChemIDplus records are missing (or at least build a list of all referenced ChemIDplus records).
• When performing a normal ChemIDplus load, it should be able to reference an existing allow-list to rule-in records that it would otherwise exclude.
• ChemIDplus should probably be able to perform a run that only adds in allow-listed records, rather than reimporting the entire source.
• We need to think about how this would work if users import both sources at once, or if they import one and then the other later.

Separate between those representing therapy concepts from those representing associated concepts.

Currently query.py queries each resource using independent search routines based off of the normalize routine for each normalizer class. After implementing the db in #29, we are now positioned to build out a generalized query using the ORM.

The normalization endpoint returns a single merged record, but it should also provide a list of merge IDs to other possible matches, if there are any. This would include any distinct merged groups that come up at the same MatchType level (should only apply to TradeName/Alias/Xref/AssociatedWith).

We should capture every time this happens with grouped concepts drawn from the same source(s) in the logs under WARNING.

We also want to run a big analysis to get an idea of where/how often this could happen.

• Run an analysis of all label_and_type fields, break out by item_type, to get an idea of how frequent alternative matches could come up (and break out by different vs same MatchType numeric values)
• log instances in real time under WARNING
• provide list of other possible merged concept IDs (probably under warnings)

Update ChEMBL ETL methods to work with DynamoDB.

(where possible)

https://docs.pytest.org/en/stable/monkeypatch.html?highlight=mocking#monkeypatching-returned-objects-building-mock-classes

• It's probably fine to pull most of the 'mock' db layer out of the existing normalization tests, as long as there's a way to catch update requests before they alter a local DB, validate them, and prevent them from making any changes.

DGIdb did not successfully identify ChEMBL concepts for several drug trade names:

• TECFIDERA
• ADRIAMYCIN
• WESTCORT
• CASODEX
• FOSCAVIR

We may not be searching those, let's be sure to add ChEMBL trade names to our search index.

Given the various changes and additions to the app's structure and features, we should update the public-facing docs to provide a more thorough introduction.

The ChEMBL normalizer returns match_type code 100 (a case-sensitive primary match) for what should be case-insensitive primary matches, eg:

...and the same for alias matches:

We will need to appropriately map terms to regimens and combination therapies.

Update DrugBank ETL methods to work with DynamoDB.

Collect and represent biosimilar linkages between drug concepts

Refactor CLI to work with DynamoDB

From: https://go.drugbank.com/releases/latest

• Declare tool version variable
• Add response date/time

Provide in both endpoints (so shelved until #62 closes)

We currently have data load methods in normalizers. We should extend these classes to allow for the full Extract, Transform, Load (ETL) process to the ORM implemented in #29.

Related issue: #7

Per https://github.com/ga4gh-discovery/ga4gh-service-info.

See https://editor.swagger.io/?url=https://raw.githubusercontent.com/ga4gh-discovery/ga4gh-service-info/develop/service-info.yaml

As we transition to a unified datastore for the application, the first objective will be to implement an ORM and define the schema for our database.

A guide on doing this with FastAPI is available here.

We already have a data model specified, which should help with constructing the initial db. As this is going to be a largely read-only, we can and should specify some useful db indices during this process.

Eg:

python3 cli.py

raises an AttributeError - should instead print the help message

Currently the application will silently load data for the normalizers in the background if missing on startup. Users are not notified, and may think application is hanging.

Possible solutions include:

1. A console handler for the logger, which can be used to emit loading messages to terminal as they are captured by the logger.
2. Raising an exception, with an alternate endpoint for preparing data.

Alongside #29, we should add migrations.
See https://alembic.sqlalchemy.org/en/latest/tutorial.html to get started.

Refactor query.py to work with DynamoDB

https://fastapi.tiangolo.com/
https://pydantic-docs.helpmanual.io/

https://www.nlm.nih.gov/databases/download/chemidplus.html

https://chem.nlm.nih.gov/chemidsearch/api

Outstanding questions:

• Some records have multiple "NameOfSubstance" attributes - could possibly resolve by "Source" tag? (currently grabbing the first item listed -- which appears to always be NLM)
• Label vs trade names - "NameOfSubstance" vs "SystematicName"?

Update Wikidata ETL methods to work with DynamoDB.