neurostuff / meta-analysis-comparison Goto Github PK

The primary goal here should be to directly compare results from NiMARE to existing libraries, in order to (1) validate NiMARE's approach and (2) potentially identify bugs in the more conceptually-based implementations (e.g., KDA).

ALE vs. GingerALE
ALESubtraction vs. GingerALE subtraction analysis
MKDADensity vs. MKDA MATLAB library
MKDAChi2 vs. MKDA MATLAB library

Are there existing tools for any of our other algorithms?

We also need to figure out what metrics we should use to evaluate tool performance.

1. Memory usage
   • How do we measure memory for an app (GingerALE) or MATLAB code (MKDA)?
2. Duration
3. Dice coefficient between thresholded and binarized statistical maps.

1. Download the GingerALE command line jar file.
2. Try the following. This should run the meta-analysis with cluster-level correction and log the time it took to run. I don't know how to measure memory usage though.

start_time=$(date +%s)

java -cp GingerALE.jar org.brainmap.meta.getALE2 \
    /Users/taylor/Documents/tsalo/ohbm2021-nimare-tutorial/data/meta-analysis/Laird2015_faces_MNI.txt \
    -mask=MNI_wb.nii -p=0.001 -perm=10000 -fwe=0.05

end_time=$(date +%s)

# elapsed time with second resolution
elapsed=$(( end_time - start_time ))
echo $elapsed

3. Run the same meta-analysis with NiMARE. We probably need to extract the MNI_wb mask for this to work.
4. Compare the cluster maps.
   • I guess calculate the Dice coefficient between the thresholded and binarized statistical maps. Is that all? How do we determine what a good result is? We could try running the correction procedure multiple times with different random seeds, but neither tool currently has a method for setting a random seed. We could hack it with NiMARE by just setting the seed before running the script, but I don't know if there's a similar solution for GingerALE.