realmarcin / sample-annotator

NMDC Sample Annotator

NMDC Sample Annotator API

Installing

pipenv

This requires python 3.7.x or later (as default python).

If you have pipenv installed:

git clone ...
cd sample-annotator
make test

venv

For those using venv, you'll need something like:

git clone ...
cd sample-annotator
python3.7 -m venv env
source ./env/bin/activate
pip install pipenv
PIPENV_IGNORE_VIRTUALENVS=1 make test

While there may be more concise ways of running commands like those below, this works:

PIPENV_IGNORE_VIRTUALENVS=1 pipenv run python -m sample_annotator.sample_annotator -R examples/report.tsv examples/gold.json

What is it?

This is a python and flask API for performing annotation of samples from semi-structured or untidy data

The API takes as input a JSON object or dictionary representing a simple sample, where each key is a metadata field

It will attempt to tidy and infer missing data according to a specified schema (currently MIxS)

Command Line

pipenv run annotate-sample --help

Usage: annotate-samples [OPTIONS] SAMPLEFILE

  Annotate a file of samples, producing a "repaired"/enhanced sample file as
  output, together with a report

  The input file must be a JSON fine containing an array of dicts

Options:
  -v, --validateonly / -g, --generate
                                  Just validate / generate output (default:
                                  generate)

  -s, --output TEXT               JSON for tidied samples
  -R, --report-file TEXT          report file
  -G, --googlemaps-api-key-path TEXT
                                  path to file containing google maps API KEY
  -B, --bioportal-api-key-path TEXT
                                  path to file containing bioportal API KEY
  --help                          Show this message and exit.

E.g.

pipenv run annotate-sample -G config/googlemaps-api-key.txt -R examples/report.tsv examples/gold.json

This will transform input such as:

[
    {
        "id": "gold:Gb0108335",
        "community": "microbial communities",
        "depth": "0.0 m",
        "ecosystem": "Environmental",
        "ecosystem_category": "Terrestrial",
        "ecosystem_subtype": "Wetlands",
        "ecosystem_type": "Soil",
        "env_broad_scale": "ENVO:00000446",
        "env_local_scale": "ENVO:00000489",
        "env_medium": "ENVO:00000134",
        "geo_loc_name": "Sweden: Kiruna",
        "habitat": "Thawing permafrost",
        "identifier": "studying carbon transformations",
        "lat_lon": "68.3534 19.0472",
        "location": "from the Arctic",
        "mod_date": "15-MAY-20 10.04.19.473000000 AM",
        "name": "Thawing permafrost microbial communities from the Arctic, studying carbon transformations - Permafrost 712P3D",
        "ncbi_taxonomy_name": "permafrost metagenome",
        "sample_collection_site": "Palsa",
        "specific_ecosystem": "Permafrost",
        "study_description": "A fundamental challenge of microbial environmental science is to understand how earth systems will respond to climate change. A parallel challenge in biology is to unverstand how information encoded in organismal genes manifests as biogeochemical processes at ecosystem-to-global scales. These grand challenges intersect in the need to understand the glocal carbon (C) cycle, which is both mediated by biological processes and a key driver of climate through the greenhouse gases carbon dioxide (CO2) and methane (CH4). A key aspect of these challenges is the C cycle implications of the predicted dramatic shrinkage in northern permafrost in the coming century.",
        "type": "nmdc:Biosample"
    },

into:

[
    {
        "id": "gold:Gb0108335",
        "community": "microbial communities",
        "depth": {
            "has_numeric_value": 0.0,
            "has_raw_value": "0.0 m",
            "has_unit": "metre"
        },
        "ecosystem": "Environmental",
        "ecosystem_category": "Terrestrial",
        "ecosystem_subtype": "Wetlands",
        "ecosystem_type": "Soil",
        "elev": {
            "has_numeric_value": 359,
            "has_unit": "meter"
        },
        "env_broad_scale": "ENVO:00000446",
        "env_local_scale": "ENVO:00000489",
        "env_medium": "ENVO:00000134",
        "geo_loc_name": "Sweden: Kiruna",
        "habitat": "Thawing permafrost",
        "identifier": "studying carbon transformations",
        "lat_lon": {
            "latitude": 68.3534,
            "longitude": 19.0472
        },
        "location": "from the Arctic",
        "mod_date": "15-MAY-20 10.04.19.473000000 AM",
        "name": "Thawing permafrost microbial communities from the Arctic, studying carbon transformations - Permafrost 712P3D",
        "ncbi_taxonomy_name": "permafrost metagenome",
        "sample_collection_site": "Palsa",
        "specific_ecosystem": "Permafrost",
        "study_description": "A fundamental challenge of microbial environmental science is to understand how earth systems will respond to climate change. A parallel challenge in biology is to unverstand how information encoded in organismal genes manifests as biogeochemical processes at ecosystem-to-global scales. These grand challenges intersect in the need to understand the glocal carbon (C) cycle, which is both mediated by biological processes and a key driver of climate through the greenhouse gases carbon dioxide (CO2) and methane (CH4). A key aspect of these challenges is the C cycle implications of the predicted dramatic shrinkage in northern permafrost in the coming century.",

Differences between input and output:

• measurement fields are normalized
• information inferred from lat_lon (currently only elev)
• TODO: ENVO from text mining
• TODO: annotation sufficiency score
• TODO: more...

Validation reports

These are created as report objects, and exported to pandas dataframes for basic statistical aggregation. See tests for details

Example report:

description	severity	field	was_repaired	category
No package specified	1			Category.MissingCore
No checklist specified	1			Category.Unclassified
Key not underscored: total particulate carbon	1		True	Category.Unclassified
Invalid field: id	1			Category.UnknownField
Alias used: total_particulate_carbon => tot_part_carb	1		True	Category.Unclassified
Parsed unit-value: 2.0 metre	1			Category.Unclassified
Missing unit 5	1			Category.Unclassified
Skipping geo-checks	0			Category.Unclassified

API Docs

TODO: readthedocs

Testing

Currently the best way to understand this code is to understand the tests

• tests
  • inputs
    • test_sample_info.yaml

This contains 'fake' samples that are intended to test validation and repair

Schema Validation

See the schema folder -- this contains a copy of the LinkML rendering of the MIxS schema from mixs-source which will later be integrated by GSC

Modules

• geo
  • currently this requires a googlemaps API key
  • TODO: rewrite to use ORNL Identify
• measurements
  • uses quantulum
  • TODO: use http://units.ontodev.com/
• text mining
  • basic repair
  • NER using fields such as study fields
• ontology
  • LinkML enumerations to ontologies

Each module will take care of different aspects

For example, the measurement module will normalized all fields in the schema with range QuantityValue

E.g. Input:

sample:
  id: TEST:1
  alt: 2m
  ...

Repair Output:

sample:
  id: TEST:1
  alt:
    has_numeric_value: 2.0
    has_raw_value: 2m
    has_unit: metre
    ...

Starting the web API

• TODO: write flask code