This repository contains some early code for the ongoing DeepNSD project, an attempt to characterize the representational structure of human visual cortex with the massive NSD fMRI dataset and a bountiful cornucopia of deep neural network models.
Our Google Colab tutorial (bit.ly/Deep-NSD-Tutorial) provides a step by step demonstration of the main functions in this pipeline, fitting the representations of a CLIP model to a subset of the fMRI data using the DeepDive package.
You can use this codebase to quickly load (in a unified API) a number of models and their associated transforms. (Please note that -- pending further development -- you will have to install the underlying packages manually.)
Models we've preprocessed include:
- the PyTorch-Image-Models (Timm) library
- the Torchvision model zoo
- the Taskonomy (visual_priors) project
- the VISSL (self-supervised) model zoo
- the Detectron2 model zoo
- ISL's MiDas models, FaceBook's DINO models...
A manuscript that details results obtained using this pipeline may be found at the reference below.
@article{conwell2022large,
title={Large-Scale Benchmarking of Diverse Artificial Vision Models in Prediction of 7T Human Neuroimaging Data},
author={Conwell, Colin and Prince, Jacob S and Alvarez, George and Konkle, Talia},
journal={bioRxiv},
year={2022},
publisher={Cold Spring Harbor Laboratory}
}