Explanation

FSharpSpreadsheetML is divided into two layers: A base layer which just wraps the OpenXML SDK methods into functions and a second layer on top which is meant to automate and ease Sheet and Spreadsheet manipulation.

Problem

At the moment, the module naming of this second layer does not reflect its purpose. Also some functions which should be in this module are spread around the project and should be bundled there.

• Setup Test Project
• ISA-XLSX
• ArcCommander

Edit: Removed FSharpSpreadsheetML from list as it was outsourced to CSBiology/FSharpSpreadsheetML

A config file could be added which stores some additional information for the ARC
This could e.g. be

• the standard editor used for querying arguments
• a default study to which all assays and computations are added per default
• github settings for syncing the arc.

This file could be created when initiating the ARC receiving its info through command line arguments

Swate makes use of the excel table objects. Therefore functions for working with these tables should be added to FSharpSpreadsheetML.

It may be beneficial to refactor the prints in the API functions to use a dedicated logger like NLog. Verbosity of commands should then be adaptable via a verbosity argument. This has the advantage that we can control both verbosity of console output and get the possibility to save these messages into a log file.

Describe the bug
When trying to edit a publication, it throws an error message after filling the fields to edit in the editor (see Screenshot).

To Reproduce
Steps to reproduce the behavior:

1. Init an ARC: arc init
2. Create an investigation: arc i create
3. Add a publication: arc i publication register -> Fill the required fields
4. Edit the publication: arc i publication edit -> Fill in the DOI -> Edit one or several field(s)

Expected behavior
Writes the edited publication into the investigation file.

Screenshots

• FSharpSpreadsheetML
• ISA-Datamodel
• ISA-XLSX
• ArcCommander

Is your feature request related to a problem? Please describe.
At the moment, the ArcCommander is only able to track files of normal file size. There is no option to track large files for git LFS. Also, the ArcCommander lacks the opportunity to clone a repository (= ARC) from a given address. Finally, it shall be possible to sparsely checkout single folders from other remote ARCs since they could have interesting stuff that a user is interested in (e.g. workflows, assay files).

Describe the solution you'd like

• An implementation of git LFS tracking for large files
• A command to directly clone repositories from a given address
• A command to sparse checkout single folders from other repositories

Is your feature request related to a problem? Please describe.
I'm always frustrated when I need to download the config file and place it somewhere.

Describe the solution you'd like
A default version of the global config file that gets created as soon as you use the ArcCommander.

Describe alternatives you've considered
The current way.

Additional context
$XDG_HOME_CONFIG would be a nice place to plant the file.

Assay files are created using the swate tool. To access this information for other tasks, a reader should be added.
As the swate tool adds tables to assay xlsx files, a prerequisite for this reader is a table reader in FSharpSpreadsheetML, which is tracked in #9.

Describe the bug
On some PCs, the notepad prompt (e.g. when using arc i create) contains text that is just printed into one single line. I.e., it misses new lines.
It is not clear on which PCs that occurs. It might be due to Windows 10 Enterprise not using UTF-8 as default encoding but ANSI.

Expected behavior
Normal new lines.

Screenshots

OS and framework information (please complete the following information):

• OS: Windows 10 Enterprise
• OS Version: 2016

Additional context
I tried forced usage of UTF-8 encoding when creating prompt file but it didn't help.

Until there is a fully clickable MacOS executable (disk image, .dmg), I would suggest to add something like the following to the installation instructions:

1. Download the latest release
2. Open a Terminal (Applications -> Utilities -> Terminal)
3. Change to directory where you downloaded the arc e.g. cd ~/Downloads/
4. Change permissions to make arcCommander executable: chmod a+x arc
5. Move arc to a place you like e.g. mv arc ~/Applications/
6. Add that place to your bash_profile to make it executable via terminal from everywhere e.g. echo 'export PATH="$HOME/Applications/:$PATH"' >> ~/.bash_profile
7. Start a new terminal for this to take effect.
8. Run arcCommander from the terminal by executing arc

MacOS security note:
On first execution, MacOS will not allow arc to be run. Instead it opens a pop-up:

"arc" cannot be opened because it is from an unidentified developer

Open the Security Panel in system Preferences (Applications -> System Preferences -> "Security & Privacy") and click the bottom-right button "Allow Anyway" right next to

arc was blocked from use because it is not from an identified developer.

Head back to the terminal and execute arc again. Another pop-up will ask you to confirm by clicking "Open".

Is your feature request related to a problem? Please describe.
Currently the ArcCommander naively ignores some errors in the arc structure. E.g. when an investigation file is missing. This can lead to some unwanted situtations. E.g. when "arc study add" is run, the study file gets initiated but the study can't be registered in the missing investigation file.

Describe the solution you'd like
Instead, prior to performing tasks that might fail halfway, sanity checks should be performed. Intuitively I would let them fail and provide additional helpful information and tips for how to solve the issue. Here I see two ways to implement this:

1. One standard check that is performed before running any command that fails if any part of the Arc specification is missing (little bit like git).
   (+) Very manageable as errors might only be done in a single place
   (-) This might fail even though the missing part might not even affect the command

2. A few selected standard checks, used in the specific scenarios they are needed.
   (+) Commands only fail if they actually can't run successfully
   (-) More complexity leaves room for more programming errors

@cgarth @JonasLukasczyk @muehlhaus
Any opinions on this?

For example if a publication does not exist, I would suggest that an update would just initiate a registration, e.g.
let

arc i publication update -d 2021_ARC05_Pub01tmp --title "Review on yeast P09 protein and its involvement
 in redox susceptibility"

do:

arc i publication register -d 2021_ARC05_Pub01tmp --title "Review on yeast P09 protein and its involvement
 in redox susceptibility"

Is your feature request related to a problem? Please describe.
The ARC specification expects a specific naming of files and folders in the arc root folder. This basic structure could be further enforced by automatically providing the user with a .gitignore reproducing these rules.

Describe the solution you'd like
A standard .gitignore should be created when running arc init.

The ac creates and uses a file called config in the default directory where config files should be located, which is ./config. The name config is inappropriate since the name is to general. This directory contains other config files as well, and it is usual to name them according to the program they configure (e.g., ParaView.config). So I suggest to rename the file to ArcCommander.config.

Problem

At the moment, writing values into the shared string table does not work. Only the number referencing the shared string will show up in the worksheet, the string itself wont be written in the shared string file.

Many functions have a normal and sst version, where the difference is that the sst versions check if the cell(s) are of type SharedString

 let getValueWithSST (sharedStringTable:SharedStringTable) (cell:Cell) =
        match cell |> tryGetType with
        | Some (CellValues.SharedString) ->

            let sharedStringTableIndex = 
                cell
                |> getValue
                |> int

            sharedStringTable
            |> SharedStringTable.getText sharedStringTableIndex
            |> SharedStringItem.getText
        | _ ->
            getValue cell

Is there a reason to now have so many sst versions of functions? The value access is guarded anyways, and will return the 'normal' cell value anyways if the CellValue is not SharedString. If this is not necessary, i would suggest to just always work with the sst versions and remove the 'normal' ones altogether to reduce namespace noise

The executable is named ArcCommander.exe, just as the respective project (ArcCommander.fsproj).
However, the Argu setup implies that the main command/executable should be arc :

let parser = ArgumentParser.Create<Arc>(programName = "arc")

When the file names should not be touched, the executable name could be changed via MSBuild properties in the fsproj file.

Describe the bug
When using arc a edit, filling out the new data, saving and exiting the editor, the console prints "could not parse query: ERROR: File was corrupted in Editor".

To Reproduce
Steps to reproduce the behavior:

1. Create new folder
2. Open console in this folder with the ArcCommander available
3. arc init
4. arc i create -> Write some data of your choice
5. arc a add-> Write some data of your choice
6. arc a edit -> Access the newly created assay with the corresponding identifier and edit some data of your choice
7. See error

Expected behavior
Normal editing of the file.

OS and framework information (please complete the following information):

• OS: Windows 10 Pro 64-bit
• OS Version: 10.0, Build 19042

Additional context
The problem already has been tracked down: It is the NewLine solution for bug #80, done in PR #81 which results in char '\013' (Carriage Return) being added to the end of each line. A deserialization of the string later fails due to incorrect pattern matching.
Solution: Add a String.replace to delete all the Carriage Returns. In non-Windows-OSs there won't be anything to replace which is no problem since the function only replaces those cases where it matches and doesn't fail if there's no matching case.

The execution of bash.sh shows a few exceptions and finally, exits on pressing any key
"parse_git_dirty:8: DISABLE_UNTRACKED_FILES_DIRTY: parameter not set "
/.../arcCommander/build.fsx (146,12)-(146,26): FSharpErrorSeverity.Warning FS1182: The value 'standardParams' is unused
/.../arcCommander/build.fsx (194,32)-(194,45): FSharpErrorSeverity.Warning FS1182: The value 'msBuildParams' is unused
Warning: Paket resolved a FSharp.Core with version '5.0.0.0', but fake runs with a version of '4.7.0.0'. This is not supported.
Please either lock the version via 'nuget FSharp.Core ' or upgrade fake.
Read fsprojects/FAKE#2001 for details.
Hint: The fake-runner has not been updated for at least 6 months. Please consider upgrading to get latest bugfixes, improved suggestions and F# features.

The ISA Datamodel and API functions were outsourced to the ISADotNet repository. Additionally the datamodel was broadened to depict the full ISA model and the API functions were adjusted accordingly.
Now the ISADotNet nuget package should be included in the ArcCommander and the ArcCommander API functions need to be adjusted too.

Describe the bug
Currently the ArcCommander still expects the metadata sheet in the assay xlsx file to be named "Investigation". This does not follow the specification defined here.

Expected behavior
Instead the ArcCommander should create and expect a metadata sheet named "assay"

Is your feature request related to a problem? Please describe.
SWATE was recently updated to version 0.5.0, including breaking changes to the headers in the assay file. To match this, ISADotNet version 0.4.0 (currently preview version) was released.

Describe the solution you'd like
The ArcCommander should now include this new ISADotNet release to ensure SWATE compatability.

Describe the bug

In ubuntu, arc i create does not open an editor, and I get the error
"could not parse query: No such file or directory"

Whereas, arc i create with the arguments works

To Reproduce
Steps to reproduce the behavior:

1. copy arc in ARC folder

2. arc init

3. arc i create

Expected behaviour
An editor should open as stated in
https://github.com/nfdi4plants/arcCommander/wiki/3.-Investigation

OS and framework information (please complete the following information):

• OS: [Ubuntu 16.04]

Additional context
Add any other context about the problem here, like ideas for a solution.

Describe the bug
When creating a new assay via arc a init or arc a add a protocols folder should be created in the assay folder, see
https://github.com/nfdi4plants/arcCommander/wiki/5.-Assay#51-addition-of-assays

Expected behavior
Create folder "protocols" during arc a init

Is your feature request related to a problem? Please describe.
ORCID (orcid.org) is the most reliable scientist ID and I would from the beginning (of ARC becoming a thing) start to invite people to use and add their ORCID (not mandatory).

Describe the solution you'd like

1. It's not part of default ISA, so maybe add key [Comment] Investigation Person ORCID to isa.investigation.xlsx.
2. And arc i person register --orcid

Is your feature request related to a problem? Please describe.
In system environments, storing and accessing config files might not be possible (e.g. when the user does not have rights)

Describe the solution you'd like
Loading the global config file could be a try function, possibly returning nothing. In this case the getConfig function could be enriched with every basic field needed for the ArcCommander to run, so there's always a file system independent fallback.

At the moment the AssayMetaData sheet gets created by the arcCommander but it is not being written to or read from. This should be implemented

Describe the bug
arc a add creates the file assay.isa.xlsx.isa.assay.xlsx.

Expected behavior
To be conform with isa.investigation.xlsx and isa.study.xlsx, replace this by

Additional context
Maybe also keep the "isa"prefix when creating a study file - e.g. "studyID_isa.study.xlsx"

Adding a new item (e.g. investigation) with duplicate title / title already existing in isa.investigation.xlsx should not just overwrite the existing item.

Reproduce error

1. arc i create --identifier "investigation_001"
2. arc i person register -l YourLastname -f YourFirstname -m YourMidInitials
3. arc i create --identifier "investigation_001"
4. Person info gone

Suggestion

Check for existing items. Respond with error "already exists" and / or suggest new name.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when I'm realizing that the ArcCommander has no Logger.

Describe the solution you'd like
A clear and concise description of what you want to happen.
A Logger. NLog seems to be an appropriate choice for our needs.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Writing an homemade Logger (not recommended if there are good ones for free, as the one mentioned above).

Additional context
Add any other context or screenshots about the feature request here.
Many other professional/famous/done-well libraries with command-line tools use Loggers. See, exemplarily, here.

Is your feature request related to a problem? Please describe.
For interoperation with the ISA-Tools, the ArcCommander should be able to export the ARC ISAXLSX Files to ISAJson.

Describe the solution you'd like
arc export maybe?
arc investigation export

Additional context

I think two major questions arise:

• Is there one command for each type of export (e.g. isa-json, ms database exports...) or might this be handled by a flag?
• What are the output objects? The whole investigation might be exported to a single investigation.json, including all info from the investigation, study and assay xlsx files. Alternatively single assays or processes or even persons might be exported.
  In this case, there could be an import and export subcommand for each object included in the ArcCommander command tree. E.g. arc study person import and arc study person export
  This might allow for easy re-use of repetitive information.

@omaus @Brilator
What do you think?

At the moment, the global config file is stored at the the same location as the ArcCommander.exe. This is not endorsed under e.g. linux, and not generally the standard way in windows.
Instead, this should be changed to the standard locations for any given os, potentially using an additional library.

For easy extension of the ArcCommander, the functions for manipulating the investigation file should be straight forward:

• add
  Add item if not already existing
• get
  If item exists, return the fields
• update
  If item exists, update the fields
• remove
  If item exists, remove it

The module and function structure in the IsaXLSX project is very messy. To improve this this project needs to be completely refactored.

• Reader
• Writer
• Transformation

Is your feature request related to a problem? Please describe.
I'm always frustrated when I want to look what's written in an ARC's investigation file but I have to open it manually in MS Excel instead of just getting a nice overview of it via the ArcCommander.

Describe the solution you'd like
There're functions for showing studies and assays. The same could be implemented for investigations.

Describe alternatives you've considered
Staying with manually opening investigation files in Excel.

I get the following error upon testing e.g. arc i create, arc i person register:

could not parse query: No such file or directory

I guess the default Mac text editor is not linked correctly?

System info:
macOS Big Sur (11.3.1)

Describe the bug
I can follow the rationale, but I'm a bit unhappy, with the handling of the study layer. e.g.

If no study identifier is given, a study is created with the assay identifier as study identifier.
This might become confusing at some point.

a) There is no isa.study.xlsx added during arc a add. So the ARC <-> isa.investigation.xlsx becomes inconsistent.
b) If I add an assay without associating a study and then choose to remove the assay, the study persists in the isa.investigation.xlsx. So I would have to remove something I have not actively added.

To Reproduce
Steps to reproduce the behavior:

1. arc a add --assayidentifier SomeTestAssayWithoutStudy
2. Check folder to find no "SomeTestAssayWithoutStudy.study.xlsx"
3. arc a remove --assayidentifier SomeTestAssayWithoutStudy
4. See isa.investigation.xlsx: study still exists.

Expected behavior
a) Add study.xlsx when out-adding study during assay creation
b) Remove study that was solely added for one assay.

OS and framework information (please complete the following information):

• OS: MacOS Big Sur
• OS Version 11.3.1

Additional context
Add any other context about the problem here, like ideas for a solution.

Is your feature request related to a problem? Please describe.
Some tools like converters will be implemented by non .Net programmers. To enable seemless integration of these tools into the ArcCommander, a standardized approach for running compiled executables should be implemented.

Describe the solution you'd like
Commands unknown to the ArcCommander could be automatically handled by checking if an executable with the same name does exist in some specific executables folder.

Since I cannot edit / comment / fork the wiki, I'll add some suggestions here that come to mind while testing arcCommander.

Many fields in the investigation file require information filled out in the assay file. This information should be easily transferrable to the investigation file.

Test, pack and upload application

This issue is meant as a place for discussing possible argument design choices

Argument Query (Asking for fields via text editor prompt)

• Add basic query functionality
• Allow user to set preferred text editor (#8)
• For Unix systems, infer editor via EDITOR (or less often, VISUAL) environment variable

I'd suggest to add -e.g. the SWATE- issue templates to this repo.

@HLWeil
@omaus

Describe the bug
Updating info via CLI does not work for me.

To Reproduce
Steps to reproduce the behavior:

1. arc i person register -f "An" -m "other" -l "person" --roles "MSc"
2. arc i person update -f "An" -m "other" -l "person" --roles "PhD"
3. Error:

...
Start Person Update
Person with the name An other person does not exist in the investigation
Done processing command

OS and framework information (please complete the following information):

• OS: MacOS Big Sur
• OS Version 11.3.1

Just putting this here to remind everyone that .NET 5 (or 6 soon) should be targeted by this tool

Is your feature request related to a problem? Please describe.
At the moment, the files created for being opened in the editor are placed in the .arc folder.

Describe the solution you'd like
Instead, the files might be placed in a more standard temp folder, like done in the htmls of Plotly.NET

Is your feature request related to a problem? Please describe.
Users might initiate an ARC in a folder where they also have other files and folders. If they later decide to delete the ARC, they have to remove all ARC-related files and folders by hand which might be a bit tricky the larger the root folder was before.

Describe the solution you'd like
An arc delete function would aid with this. It can be easily implemented since we always know about every possible folder and file in an ARC root.

Describe alternatives you've considered
Let the users still delete ARC-stuff manually.

Additional context
A functionality with such consequences will be located in the Danger Zone, thus there should be a query if the action shall really be performed or not (typically, a y/n scenario).