Explorations of using a convolutional neural network to understand brain activations during movie watching
Psychoradiology · 07.02.2024 · 4
DOI: 10.1093/psyrad/kkae021 ↗Kurz gefasst
Fachartikel in der Zeitschrift Psychoradiology, online erschienen am 04.11.2024. Verfasst von Wonbum Sohn und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NIMH NIH HHS, National Institute of Mental Health.
Abstract (Originalsprache)
Background
Naturalistic stimuli, such as videos, can elicit complex brain activations. However, the intricate nature of these stimuli makes it challenging to attribute specific brain functions to the resulting activations, particularly for higher-level processes such as social interactions.
Objective
We hypothesized that activations in different layers of a convolutional neural network (VGG-16) would correspond to varying levels of brain activation, reflecting the brain's visual processing hierarchy. Additionally, we aimed to explore which brain regions would be linked to the deeper layers of the network.
Methods
This study analyzed functional MRI data from participants watching a cartoon video. Using a pre-trained VGG-16 convolutional neural network, we mapped hierarchical features of the video to different levels of brain activation. Activation maps from various kernels and layers were extracted from video frames, and the time series of average activation patterns for each kernel were used in a voxel-wise model to examine brain responses.
Results
Lower layers of the network were primarily associated with activations in lower visual regions, although some kernels also unexpectedly showed associations with the posterior cingulate cortex. Deeper layers were linked to more anterior and lateral regions of the visual cortex, as well as the supramarginal gyrus.
Conclusions
This analysis demonstrated both the potential and limitations of using convolutional neural networks to connect video content with brain functions, providing valuable insights into how different brain regions respond to varying levels of visual processing.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA.
- Rutgers Biomedical and Health Sciences, Rutgers School of Graduate Studies, Newark, NJ 07039, USA.
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, 51806, China.
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518060, China.
ORCID-Profile: Xin Di, Zhen Liang
Förderung
- NIMH NIH HHS
- National Institute of Mental Health
Zitieren
APA
Sohn, W., Di, X., Liang, Z., Zhang, Z., & Biswal, B. B. (2024). Explorations of using a convolutional neural network to understand brain activations during movie watching. Psychoradiology, 4. https://doi.org/10.1093/psyrad/kkae021
Vancouver
Sohn W, Di X, Liang Z, Zhang Z, Biswal BB. Explorations of using a convolutional neural network to understand brain activations during movie watching. Psychoradiology. 2024;4. doi:10.1093/psyrad/kkae021
BibTeX
@article{Sohn2024,
author = {Sohn, Wonbum and Di, Xin and Liang, Zhen and Zhang, Zhiguo and Biswal, Bharat B},
title = {Explorations of using a convolutional neural network to understand brain activations during movie watching},
journal = {Psychoradiology},
year = {2024},
volume = {4},
doi = {10.1093/psyrad/kkae021}
}
EUVIMED