Add more integrity checks on fields in JSON schema 2 about nmdc-schema

[ssarrafan]

Continuation of work discussed (at least on GH) from the May sprint at https://github.com/microbiomedata/nmdc-metadata/issues/308

[ssarrafan]

Third level of checks handled in https://github.com/microbiomedata/nmdc-metadata/issues/362

[ssarrafan]

Removing other assignees. @turbomam let me know if this assignment isn't you.

[turbomam]

RE Is the ID prefix valid? (e.g. KEGG.KO vs KEGG.ORTHOLOG)

I see prefix definitions, especially in nmdc-schema/src/schema/annotation.yaml

MAM@MAM-M74 schema % pwd
/Users/MAM/Documents/gitrepos/nmdc-schema/src/schema
MAM@MAM-M74 schema % grep -i kegg *
annotation.yaml:      - KEGG.PATHWAY
annotation.yaml:      - KEGG.REACTION
annotation.yaml:      - KEGG.ORTHOLOGY  ## KO number
core.yaml:      - KEGG.COMPOUND

Anywhere else I should be looking? @wdduncan @cmungall

[turbomam]

RE Is the local part of the ID syntactically conformant? (e.g. KEGG:K\d+)

I don't see patterns for the local parts, at least not in nmdc-schema/src/schema/annotation.yaml

  pathway:
    aliases:
      - biological process
      - metabolic pathway
      - signaling pathway
    is_a: functional annotation term
    description: >-
      A pathway is a sequence of steps/reactions carried out by an organism or community of organisms
    slot_usage:
      has_part:
        range: reaction
        multivalued: true
        description: >-
          A pathway can be broken down to a series of reaction step
    id_prefixes:
      - KEGG.PATHWAY
      - COG
    exact_mappings:
      - biolink:Pathway

[turbomam]

very rough example for nmdc-schema/src/schema/annotation.yaml from @cmungall

functional annotation term:
    aliases:
      - function
      - functional annotation
    is_a: ontology class
    slot_usage:
      id:
        pattern: "^(KEGG.ORTHOLOG:K\\d+|EC:\\d+\\.ETC)$"
    description: >-
      Abstract grouping class for any term/descriptor that can be applied to a functional unit of a genome (protein, ncRNA, complex).
    abstract: true
    todos:
      - decide if this should be used for product naming

[turbomam]

Was microbiomedata/nmdc-metadata issue 360

I will be adding local part patterns to the yaml files in this repo.

@ssarrafan @wdduncan @cmungall

[turbomam]

See notes from @cmungall at PR #70, especially

I suggested the parens to indicate that we need other IDs, e.g (FOO|BAR|...)

[turbomam]

@wdduncan and @cmungall : I can't find patterns for COG or RetroRules at the BioRgistry, or sample usages in the MongoDB.

example working query:

> db.raw.functional_annotation_set.find({"has_function": {"$regex": "^pfam", $options: 'i'}})
{ "_id" : ObjectId("6011a09275ead576bdc24c02"), "subject" : "nmdc:Ga0482148_260452_3_287", "has_function" : "PFAM:PF00001", "was_generated_by" : "nmdc:8a43ec3baf8aafe09d96eb7fbf58c916" }
{ "_id" : ObjectId("6011a0d2666867f660864500"), "subject" : "nmdc:Ga0482235_197390_1_279", "has_function" : "PFAM:PF00001", "was_generated_by" : "nmdc:e763e255fa74e2629d7d86e10f838d4b" }
{ "_id" : ObjectId("6011a1113350938c11bd6527"), "subject" : "nmdc:Ga0482263_74753_2_277", "has_function" : "PFAM:PF00001", "was_generated_by" : "nmdc:686818cb31dc45d3d4482847ec007584" }

But neither of these return any matches:

• db.raw.functional_annotation_set.find({"has_function": {"$regex": "^cog", $options: 'i'}})
• db.raw.functional_annotation_set.find({"has_function": {"$regex": "^retrorules:", $options: 'i'}})

[turbomam]

@ssarrafan do you have a sense of who raised this concern? Can I close it?

[turbomam]

It doesn't seem like it's really specific to checks on the contents of a JOSN file by the JSON schema serialization of the schema.

[turbomam]

Is the concern especially about validating KEGG-related CURIes?

[ssarrafan]

@ssarrafan do you have a sense of who raised this concern? Can I close it?

This is from 2021 so I don't remember which meeting this came from. I would say it can probably be closed.