afids / afids-validator

Validator for the anatomical fiducial placement protocol

Anatomical fiducials (AFIDs) is an open framework for evaluating correspondence in brain images and teaching neuroanatomy using anatomical fiducial placement. The AFIDs Validator project aims to build a web application that allows the user to upload an FCSV file generated using the AFIDs protocol, and validate that it conforms to the protocol.

afids-validator (https://validator.afids.io)

Development

poetry is used to manage dependencies. To install, run the following command:

curl -sSL https://install.python-poetry.org | python3 -

For detailed setup instructions, see the documentation here.

Once installed, you can set up your development environment by:

1. Git clone the afids-validator repository git clone https://github.com/afids/afids-validator.git
2. Set up python environment via poetry shell
3. Install the required libraries via poetry install --with dev
4. Install the pre-commit action via poetry run poe setup. This will automatically perform quality tasks for each new commit.
5. Update configuration in .env.template and rename to .env file

This will allow you to make changes and perform the necessary formatting and linting tasks. To test changes, the easiest way is via docker compose. To use this, you will need to install Docker.

Once installed, you can run docker compose up --build in the terminal.

If there are no errors, you can test it out locally at http://localhost:5001

After you are done testing, you can hit CTRL/CMD+C on the terminal to exit out of the instance and run docker compose down to remove unused containers.

Testing afids upload

If docker compose successfully starts the required services, you will first want to enter the afids-validator container interactively to migrate the database, enabling testing of database uploads.

# Enter the container interactively
docker exec -it afids-validator-afidsvalidator-1 bash

# Migrate the database
flask db upgrade -d /usr/local/lib/python3.9/dist-packages/migrations/

Note: You may need to change the container name (i.e. afids-validator-afidsvalidator-1) accordingly.

Testing login

To test the login with ORCID iD:

1. Create an account (with a mailinator.com email address) on sandbox.orcid.org
2. Follow these instructions to get a client ID and client secret. Set the Redirect URIs to your local testing address (eg. 127.0.0.1:5001, localhost:5001)
3. Update your local .env file with your new credentials.
4. Locally change the URLs in afidsvalidator/orcid.py to start with api.sandbox.orcid.org
5. Run the application and test your login.

afids-validator's Issues

Internal server error

The problem

I tested the afids protocol and while using the validator I am getting internal error

Environment

• Safari (13.1.1)

_

Validator Logic (Do you mean section?)

We want the validator to fail completely in some cases and partially in others.

Completely:
incorrect # of fiducials
Some tolerance failure if X partial failures, fail completely (~25%)
misordering of the fiducials inside the files

Partially:
Incorrect capitalization
Incorrect spelling
L-R fiducial

When throwing a complete failure still throw out as many errors as possible.
Any thoughts @jclauneuro @tkkuehn

[Bug] Upload Error

Upload of fcsv throws error with message below:

Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Unit test isn't catching error

@tkkuehn - please add log

Validate given json object

support for 3D Slicer v4.11 and beyond

The syntax for headers in 3D Slicer v4.11 has changed resulting in some erroneous behaviour if loading files into 3D Slicer with an earlier version (images in the same space are flipped relative to each other -- one is RAS and the other LPS).

Some details here: https://discourse.slicer.org/t/converting-fiducial-coordinates-from-ras-to-lps/9707/6

For now, the simple thing to do is enforce the use of 4.10.2 for afids annotations (we've changed all the documentation accordingly). At some point (time permitting), dealing with the change in header can be dealt with by the validator.

Update README / Add instructions

An update should be / needs to be made with the new instructions with the (future) migration to heroku. There are screenshots on the mattermost with detailed steps, just needs to be added.

dataset comparison and outlier detection

Assuming anatomical similarities between subjects (which most of the points are?), could be used to evaluate registration.

Otherwise if fiducial placement + registration is assumed reliable, could be used as coarse measure of anatomical differences between groups (i.e. patients vs controls)

Fiducial Flip - Coordinate Systems

The problem

Need to rewrite the check for fiducial flipping to be version agnostic Original implementation defaulted to "RAS" coordinate system (valid for version 4.10 and below). Upcoming implementations to default to "LPS" coordinate system (which may still be the case in version 4.10 and below). Will need to check the header file. Possible header information include the following:

RAS - "0" or "RAS"
LPS - "1" or "LPS"

Environment

N/A

PR template was adopted from appium

Valid Extensions

Current valid extensions are .txt, .csv, and .fcsv files. Are .txt files going to be valid moving forward (what was original reason .txt files were valid)?

BUG: Mismatched fcsv column

The problem

Moving from #91 to separate issue. Validator throws a server error when the following fcsv is uploaded: 2009bMNI_T1_TM_01.txt.

There appears to be a mismatch of columns / missing information for first two fiducials (AC. PC). See below for the two problematic lines.

vtkMRMLMarkupsFiducialNode_0,0.023,-8.165,-16.415,0.000,0.000,0.000,1.000,1,1,1,AC,,vtkMRMLScalarVolumeNode1
vtkMRMLMarkupsFiducialNode_1,0.536,-25.470,-2.099,0.000,0.000,0.000,1.000,1,1,1,PC,,vtkMRMLScalarVolumeNode1

Note the two commas following label description (AC, PC) and missing label (1, 2).

NA: To discuss potential implementation of mismatched columns. Currently, this error bypasses current checks. Correct number of columns, but label description was in the place of label, and description is empty.

Environment

• Browser (with version): Safari (13.1.1) and Chrome (83.0.4103.97)
• Dropdown menu option: Any option
• File uploaded (if applicable): 2009bMNI_T1_TM_01.txt.

PR template was adopted from appium

css style integration

Just added a basic MIT license CSS style (normalize.css).

TODO: provide link in input and output html files.

Heroku App Name

The problem

Not really code related, but more app related. We should consider renaming the heroku app from fidvalidator to afids-validator to align with the naming of other pages and repos.

Environment

N/A

PR template was adopted from appium

Branch clean up

Should go through and removing branches that have been merged in or are stale.

Fiducial subsets

One idea in the past few weeks was to implement the handling of a subset of fiducials (which may be better for teaching purposes). In addition to this, I believe a feature that was implemented in the previous brainhack was to accept fiducials even if they are not in the correct order.

A suggestion is to have an additional checkbox that ignores the number of fiducials in the files and only check for matching descriptions.

Will have to think about this one a little bit more as it may affect content validation (#71)?

.fcsv structure validation

A feature that is nice to have, especially when testing new fiducials that don't match existing templates or for new users. Idea here is to validate the structure of the fcsv without any actualy computation of error. Data should still be (potentially) stored in order to keep the fiducials points for future storage in the event the template is to be created.

My understanding is that currently, the blank entry in the drop down menu does not perform a comparison. Maybe we can remove the blank entry and replace it with a default "Structure Validation" or something. Not how to best move forward with this.

@Park-Patrick @tkkuehn

Deal with file uploading

wtforms handles all the file upload logic, we just want to read the uploaded file and pass it to csvtojson, so that we don't need to save it to the hard drive

Deploy from released version

The problem

Currently validator is deploying from master upon every PR merge. Want to change it such that github actions deploys on a new version release.

Environment

N/A

PR template was adopted from appium

unique identifiers

Some thoughts brought up today were:

• email
• full login capabilities
• security
• time to implement

My tentative suggestion is to have a userid (i.e. jsmith5) field that the uploader can fill in before adding data. This would mean that anyone can add to anyone's data collection, but I think we can assume some goodwill here for now.

Any uploaders not interested in this function can simply use "guest" to have the same experience as they do now.

Pros

• no storage of sensitive information (email / password)
• low implementation cost
• gives us the ability to separate analytics triggers from uploads (allows other features to build out)

Cons

• high tamper-ability

Happy to hear everyone's thoughts!

AFIDs Class / Internal Representation

The problem

We currently parse .fcsv files directly into JSON strings. There's not really any good reason to do it this way, and now that Slicer is moving from .fcsv files to .json files, we should have an internal representation that can import from and export to either file format. An AFIDs class seems like a good approach.

BUG: JSON fiducial order

The problem

The order of the fiducials when it is saved matters for the JSON file. Currently parsed by index (no other key to delineate between different fiducials) before checking labels. Not sure if there is a way around this, but if not we should implement a check to ensure they are in the correct order!

This can be tested by converting the valid_flip.fcsv to a json file and running it through validator on the afids-class branch.

Environment

• Browser (with version): N/A
• Dropdown menu option: N/A
• File uploaded (if applicable): Converted valid_flip.fcsv to valid_flip.json with latest Slicer (one in test directory has been manually fixed)
• Branch: afids-class

PR template was adopted from appium

UX Optimization Outline

Redesign of website for user-friendly experience. This issue includes:

• Outline
• Standardization of web pages

Explain uploading to database

The problem

Right now, a novice user who uses the database might be confused by the "Upload to Database" checkbox on the validator page. The purpose of the database and why they may want to upload their fiducial set to the database does not seem to be explained anywhere.

Adding an explanation somewhere on the site might improve the user experience.

PR template was adopted from appium

Visualization / Stats

Histogram of errors

• Histogram of errors in relation to cartesian coordinates (i.e. bias)
• subplots?
• mean squared error (are you doing better in certain sections? left vs right)
• top 3 best / worst?

Switching to json?

Slicer5 is moving towards saving markups as json files. This has many advantages:

• more metadata can be saved (for each point, or for the entire list)
• multiple fiducial lists can be saved in a single file
• various markup types can be saved in a file (not just points but lines, curves, planes, etc.)
• json schema file can be created, which can be used for automatic validation of the file (presence of certain labels, values, etc.)
• unlike csv, json is very well defined file format (no questions about what separators are used - comma, semicolon, tab; or how to escape quotation marks and separator characters; or how to encode international characters, etc.)

We would keep supporting loading of .fcsv files and for now fiducials are still saved in .fcsv file by default, but we would want to transition to .json format as much as possible to take advantage of all the features that it offers.

What do you think about this transition?
What tools do you currently use to edit or process fcsv files?
How difficult would it be to update your toolchain to work with json files instead of/in addition to fcsv?

We can discuss this here, but maybe the best would be if you could join this discussion:
https://discourse.slicer.org/t/markupsline-fcsv-loads-as-markupsfiducials/11545/33

Template (answer key csv) selection

• Could be from drop down menu
• Correct CSVs stored on repo with references to path

Refactor/Lint Codebase

The problem

A lot of the code is qualitatively messy. Would be nice to run the whole thing through pylint or something and refactor it to clean it up a bit.

Add github CI for tests

Get circle CI or some such to run the tests for json validation.

Submodule / fcsv reorganization

Current use of submodules causing usage of half of the slug size. Remove the current submodule usage and add the necessary fcsv files separately - will likely also require modification of code to read file from appropriate directory.

Dropdown menu

Just noticed that after selecting one of the templates in the dropdown, the name of the template selected doesn't appear. This is likely just a color issue and if we can change the font colour to white it will likely solve it.

Ultimately a UI/UX thing, but as this is currently deployed on the live website, might be a good idea to fix it soon.

Code Cleanup

This kind of goes along with everything else, but at some point need to go through and clean up the code. Been making some changes to the website and there are a lot of redundant / unused imported modules. A good target might be to have this cleaned up for OHBM.

Fiducial Names

Mentioned in a previous PR.

Some fiducials names from template in sub-module are mismatched with protocol (@jclauneuro)

Fiducials affected:

• culmen -> Culmen
• intermammillary sulcus -> Intermammillary sulcus
• genu of CC -> Genu of CC
• splenium -> Splenium of CC

Templates affected:

• sub-Agile12v2016
• sub-Colin27
• sub-MNI2009cAsym
• sub-PD25

Add contributions document

The problem

Currently repository does not have a contributions document for users who may want to contribute. This would be a useful thing to have.

PR template was adopted from appium

Validator Compare page: dropdown menu and disambiguation

Consider providing a link that provides information about the validated templates included in the validator so far.

Provide better descriptors in the dropdown menu (currently uses the name of the associated .fcsv file). Also by default the validator checks an input .fcsv file remaining agnostic to template choice, so perhaps the associated text should be "Check format of .fcsv file only" (or something to that effect).

multi fcsv upload

discussed as fairly self-explanatory feature.

Add ability to upload / validate multiple fcsvs at once. seems reasonable to restrict one template per upload.

Landing Page Intro

Write up for landing page + pictures/video/demo, etc.

Heroku Deployment

The problem

Something to make life easier in the future - set up a Github actions workflow to automatically deploy to Heroku when a PR is merged.

Something similar to this: https://github.com/marketplace/actions/deploy-to-heroku

This will require someone with access to this repo's settings to add in the login and password to the Github secrets so we can make use of it in the workflow!

Environment

• Github Actions
• Heroku

PR template was adopted from appium

Website Bugs

A couple of minor issues:

1. The alignment seems to be a little bit off (with new site pushed). You can see it in the screenshot below. It also appears on one of the other pages.

2. Guess we forgot to take out the copyright on the bottom there, but can do that as part of the code cleanup!

Remove View DB and Analytics buttons

Opening up a new issue to remove the "View DB" and "Analytics" buttons from public web page until those features are ready to be implemented.

BUG: Actions linting triggered on merge

The problem

Github actions workflow currently triggering on merge, which should not happen.
Look into conditionals!

Environment

N/A

PR template was adopted from appium

Replace Visualization Code

Discard old visualization code and implement with new framework

Automate file reading to populate json object

• From reading in file (fcsv, csv) to populating json object
• fid name, x, y, z coordinates,

manage traffic / uploads

• Change data upload to db as opt-in checkbox (to prevent being flooded with nonsense entries)

• Someone who chooses not to upload should still be able to see metrics / visualization (temp entry?)

• This could also be a good place to categorize traffic, i.e. (Is this your first time using the afids protocol)

Thoughts? @tkkuehn and others

Time Zone

The problem

Noticed after uploading file with time stamp, time zone was off by 4 hours. Digging into this, Heroku uses default UTC, while testing used local time zone. Should indicate in the timestamp the time zone.

Environment

N/A, on website

PR template was adopted from appium

NHP vs Human Toggle

The problem

To be clearer to the end user and ensure the database stays clean we need to ask the user if the data they are uploading is NHP or human. This can be done in a toggle or dropdown. Mixing organism and template should result in a warning.

PR template was adopted from appium

Update document website

In the short term, the documents page should be updated to the new netlify link.

validation of non-human template file structure

Creating this issue so that it can be tracked.

The problem

per @jclauneuro "Validate .fcsv structure template" was not working with no error messages. Seemed to work with human templates. I have added a non-human template for testing, and the non-human template appears to be validated properly.

Would you be able to provide some additional information @jclauneuro?

Environment

as tested by @kaitj

• Browser (with version) - Chrome v83.0.4103.61 (64-bit)
• Dropdown menu option - "Validate .fcsv structure template"
• File uploaded (if applicable) - "macaqueMNI_mean.fcsv" (renamed as valid_nhp.fcsv)

PR template was adopted from appium

update README.md with python anywhere integration instructions

Compute distances between client json values and template json

• call csv to json function for template when requested
• run euclidean distance calculation between corresponding array elements

OHBM hackathon ideas

Inspired by @kaitj's resolution to clean up the code over the coming months. Just wanted to open an issue to explore how we might try to engage the greater OHBM community at the upcoming meeting in Montreal.

Perhaps ideas for either tutorials ("train track") and/or extensions to the current AFIDs-validator workflow.

Update protocol link

The Problem

Need to update the protocol link to use the GitHub.io url after the current domain expires. Believe this is suppose to happen next week.

Environment

• Browser (with version)
• Dropdown menu option
• File uploaded (if applicable)

PR template was adopted from appium