nfdi4plants / arctokenization Goto Github PK

Just so i don't forget about this:

We definitely need more unit tests, especially integration tests since a we have some functionality tested at the utmost atom level but this does not help us if the whole pipeline functions cannot parse an, e.g., Investigation file.

example:

we need some way of grouping the flat token list to column-based objects, e.g. persons.

First idea would be using cell address information.

see https://github.com/nfdi4plants/ISADotNet/blob/arctrl/src/ISA/ISA/Regex.fs

The changes seem fine for getting just a CvParam list. But don't forget to extend the list of KeyMappers if you want to transform all fields to CvParams. I didn't finish it up completely.
https://github.com/nfdi4plants/ArcGraphModel/blob/74c57e61aa5cd798cf0702f9daab053b14af32b2/src/ArcGraphModel.IO/ISA/KeyParser.fs#L38

Originally posted by @HLWeil in #17 (comment)

Edit by kMutagene

Things that need to happen for this (for structural terms in metadata sheet sections):

• Port obo parser to separate library, reference it (done, see https://github.com/CSBiology/FsOboParser)
• Create structural ontologies for
  • Investigation metadata sheets
  • Study metadata sheets
  • Assay metadata sheets
• Create CvParams from all terms in a structural ontology
• Add these terms to the matching logic that creates the flat token lists for metadata sheets

Currently the column names are implemented in at least 3 different libraries. This should be unified.

As far as I understand it, the ArcGraphModel is the bottom most library everything is based on.

I suggest, to implement the column names as union type with member functions for qol features.

See example here in Swate:

https://github.com/nfdi4plants/Swate/blob/177348ab6b27f51e6aee43c007c7396063b9b52a/src/Shared/OfficeInteropTypes.fs#L27

When, e.g., taking a CvParam and calling printfn "%A" on it, it does not display the same as when calling its .ToString() method.
Fix this.

But it would definitely be better if it was parsed as CvParam. Maybe allow Comments at all?

Also: ORCID misses a CvTerm in the respective ontology.

Le workaround...

let cvparams = 
    params      // this is a IParam list, parsed from an ISA.XLSX file
    |> List.map (
        fun p -> 
            match CvParam.tryCvParam p with
            | Some cvp -> cvp
            | None -> CvParam(p.ID, p.Name, p.RefUri, p.Value, p :?> CvAttributeCollection)
    )

For arc-validates CvParam approach, we need a new parser that only takes CvParams instead of CvParams and some CvParams already parsed into CvContainers. Therefore, the parser functions in https://github.com/nfdi4plants/ArcGraphModel/blob/main/src/ArcGraphModel.IO/ISA/Worksheet.fs need to be updated for using convertTokens function instead of parseLine.

https://github.com/nfdi4plants/ArcGraphModel/blob/74c57e61aa5cd798cf0702f9daab053b14af32b2/src/ArcGraphModel.IO/ISA/Worksheet.fs#L24-L37

The name is parseColumn but it parses the columns of the worksheet's first table while

https://github.com/nfdi4plants/ArcGraphModel/blob/74c57e61aa5cd798cf0702f9daab053b14af32b2/src/ArcGraphModel.IO/ISA/Worksheet.fs#L39-L52

parses the columns of the whole worksheet, ignoring tables though the name is parseTableColumns.

This means, the names are mixed up.

after the big updates surrounding arctrl, there will also be a new, more flexible input/output column syntax. See this issue:

nfdi4plants/Swate#316 (comment)

Since we partition the columns into IO vs rest, we just need to update our regexes accordingly, so let this issue stand here as a reminder to do that eventually.

Might be a bit more difficult since CvParams can be nested infinitely via having CvParams in their Attribute collection (therefore recursion is needed)...

Atm., a lot of the tokenization already packs some information into CvAttribute collection among other things. There are also some Bugs in parsing Keys associated with this.
We need to disentangle this structural muddle and also add some more testing to make this mess more stable than it is now.