ziqiren / hyper

HYPER (brain decoding framework) 🧠 + 🤖 + 📖 = ✨

This repo accompanies the original paper "Hyperrealistic neural decoding: Reconstructing faces from fMRI activations via the GAN latent space" (Dado et al., 2020 (biorxiv)). This study introduces a novel experimental paradigm that uses synthesized yet highly naturalistic stimuli with a priori known feature representations together with an implementation thereof for HYperrealistic reconstruction of PERception (HYPER) of face images from brain recordings. The goal was to reveal what information was present in the recorded brain responses by reconstructing the original face stimuli presented to the participants.

The experiment

Two participants were looking at images of faces while we recorded their brain responses in the MRI scanner. After that, we trained a model to reconstruct what the participants were seeing from their fMRI recordings alone.

The magic ingredient

The faces in the presented photographs do not really exist, but are artificially generated by a progressiveGAN (PGGAN) from latent vectors. As it turns out, the PGGAN latent space and the neural face manifold have an approximate linear relationship that can be exploited during brain decoding. That is, the latent vectors used for face generation effectively capture the same defining stimulus features as the fMRI measurements. As such, we can predict the latents that underlie the perceived face images and feed them to the PGGAN for (re)generation, leading to the most accurate reconstructions of perception to date.

🤖🤖🤖

Required components

This repo contains (demo) code in Jupyter Notebooks to present the approach. All required data to reproduce the results, preprocessing steps and test set images (stimuli and reconstructions) are made available on Google Drive.

Conclusion 🚀

For the first time, we have exploited the latent space of a generative model in neural decoding during perception of synthetic face photographs. It should be noted that the results of this study are valid reconstructions of visual perception regardless of the nature of the stimuli themselves. Considering the speed of progress in the field of generative modeling, this framework will likely result in even more impressive reconstructions of perception.