neuroscout / neuroscout-paper Goto Github PK

get number of unique subjects and average scan time per subject for all datasets

Upload for A25Bv (frequency model for LTS) fails with following traceback (using neuroscout-upload):

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/pyns/api.py", line 114, in _make_request
    resp.raise_for_status()
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: UNPROCESSABLE ENTITY for url: https://neuroscout.org/api/analyses/A25Bv/upload

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/bin/neuroscout", line 11, in <module>
    load_entry_point('neuroscout-cli', 'console_scripts', 'neuroscout')()
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/neuroscout_cli/cli.py", line 65, in main
    command(deepcopy(args)).run()
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/neuroscout_cli/commands/upload.py", line 8, in run
    return super().run(upload_only=True)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/neuroscout_cli/commands/run.py", line 112, in run
    n_subjects=n_subjects)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/pyns/models/analysis.py", line 277, in upload_neurovault
    n_subjects=n_subjects, collection_id=collection_id)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/pyns/api.py", line 121, in _make_request
    raise requests.exceptions.HTTPError(error)
requests.exceptions.HTTPError: 422 Client Error: UNPROCESSABLE ENTITY for url: https://neuroscout.org/api/analyses/A25Bv/upload

Once this is solved, #6 can be merged.

Logo on https://neuroscout.github.io/neuroscout-paper/intro.html is broken

• Identical DMs show up for different models
• hierarchical structure is messed up
• missing effect and variances

@tyarkoni will revise list of single predictor models to add "body" and remove a few.

This is just a record of some "quirks" that are dataset or feature specific. Feel free to edit and add to the list
Will work on making a more general version of this for public consumption

• StudyForrest is in German and has no ingested speech transcript
• Sherlock movie is present in the Sherock and and SherlockMerlin datasets, with different subjects. We're going to focus on the Sherlock dataset for the Sherlock task, and the Merlin task from the SherlockMerlin dataset.
• SchematicNarrative has no "tokenized" BERT features because there are independent stimuli within each run.
  See: neuroscout/neuroscout#772
• Life dataset has no faces due to the nature of the stimuli

remove feature descriptions for features we do not use (e.g., BERT or AudioSet)

For maximal reproducibility

To fix discrepancy w/ task names

Points maybe worth mentioning

• Neuroscout makes multi-dataset reproducible workflows accessible
• Makes using novel features easy
• Not mutually exclusive w/ experimental research
• Caveats in the interpretation of results (e.g., features are model dependent)
• What next
  • More datasets and features
  • Dataset release
  • Enable browsing
  • Better integration with meta-analysis workflow
  • Support other models?

create utils for surface plots (at least for single-predictor models), we need more compact ways to visualize results

re-run some audioset meta-analyses w/ new datasets (e.g., speech, music, whistling?), possibly with thresholding

Once the set of models is final, we should re-run all notebooks so to include at least some sample reports and timeline/distribution plots for regressors of interest.

simpler figure for platform overview (very "functional", displaying Neuroscout components by their role in the workflow)

Datasets with no model (e.g. studyforrest for entropy models) are still included in the collections as empty nodes (studyforrest: {}) We should drop from all collections all nodes for datasets with no associated analysis.

we haven't pushed meta-analytic maps to this repo (though they can be reproduced).
should we do that?

• Get surprisal metrics for different models and across transcript vs. force-aligned (= no punctuation)
• Look at correlations between models
• Qualitative inspection of examples
• For now, only focus on window_size = 25

• Export to Overleaf
• Add Acknowledgments
• Table 1: Add links
• Fig 3 label is on dotted line
• Add line numbers (perhaps make 2 version at last minute)
• Consistent use of em dash (—). It should be specifically that charachter and have no spaces around it.
• The FaceNet methods section has several variables not presented (e.g. first_time_face)
• Rebuild jupyter book and link

Citation related issues:

• fMRIprep section needs references properly cited
• Manually check all refs
• Markiewickz citations look odd to. e.g.: C. Markiewicz et al., 2021; C. J. Markiewicz et al., 2021 - statsmodels and fitlins respsectively

fit separate models with re-extracted mel features to reconstruct tonotopic maps (not necessarily relevant for the paper, but as preliminary result for ohbm submission and to kick-start some audio analyses).
Analysis should probably be set up as classification.

Single predictor models:

• Should be all done

Frequency analyses:

• We need to look into which models to report, but probably something incremental
• Run on NNDB and Narratives
• Check consistency in estimators and if inconsistent rerun

Shot-change:

• Re-run on NNDB
• Check consistency in estimators and if inconsistent rerun

Lancaster norms:

• Rerun on NNDB and Narratives
• Check consistency in estimators and if inconsistent rerun

FaceNet:

• @adelavega extracts NNDB
• Run NNDB
• Check consistency in estimators and if inconsistent rerun

AudioSet:

• Run music on NNDB and Narratives (need to be extracted)
• Maybe explore couple of other features (selectively pick them from the ontology)
• Check results and make decision on whether to keep them
• If we keep them, check consistency in estimators and if inconsistent rerun

BERT:

• Drop if for now (play with it in separate projects, e.g., https://github.com/rbroc/eval-neuroscout-lm) '
• Drop the embeddings too, but that could also go in a separate project

Reading brain dataset:

• Ignore for now, but maybe run frequency on it after everything else is done.

make figure explaining feature extraction for a number of clarifai predictors (display top/bottom frames)

especially:

• shorten FFA and frequency paragraphs
• add Lancaster norms discussion

In some analyses that were re-run w/ afni, the original imagines without space- entity (typically run w/ nilearn) were not overwrittten.

Thus they were uploaded again to the new afni only collection.

Detect and correct these NV collections.