EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data

Yuda Bi iD, Anees Abrol iD, Zening Fu, Vince D Calhoun

Human Brain Mapping · 2024-12 · 45 (17)

DOI: 10.1002/hbm.26783 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Human Brain Mapping, online erschienen am 26.11.2024. Verfasst von Yuda Bi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia Tech, Emory, Atlanta, Georgia, USA. Hauptthemen laut MEDLINE: Schizophrenia, Magnetic Resonance Imaging, Deep Learning, Multimodal Imaging. Die Arbeit wurde bisher 59-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal neuroimaging is an emerging field that leverages multiple sources of information to diagnose specific brain disorders, especially when deep learning‐based AI algorithms are applied. The successful combination of different brain imaging modalities using deep learning remains a challenging yet crucial research topic. The integration of structural and functional modalities is particularly important for the diagnosis of various brain disorders, where structural information plays a crucial role in diseases such as Alzheimer's, while functional imaging is more critical for disorders such as schizophrenia. However, the combination of functional and structural imaging modalities can provide a more comprehensive diagnosis. In this work, we present MultiViT, a novel diagnostic deep learning model that utilizes vision transformers and cross‐attention mechanisms to effectively fuse information from 3D gray matter maps derived from structural MRI with functional network connectivity matrices obtained from functional MRI using the ICA algorithm. MultiViT achieves an AUC of 0.833, outperforming both our unimodal and multimodal baselines, enabling more accurate classification and diagnosis of schizophrenia. In addition, using vision transformer's unique attentional maps in combination with cross‐attentional mechanisms and brain function information, we identify critical brain regions in 3D gray matter space associated with the characteristics of schizophrenia. Our research not only significantly improves the accuracy of AI‐based automated imaging diagnostics for schizophrenia, but also pioneers a rational and advanced data fusion approach by replacing complex, high‐dimensional fMRI information with functional network connectivity, integrating it with representative structural data from 3D gray matter images, and further providing interpretative biomarker localization in a 3D structural space.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

SchizophreniaMagnetic Resonance ImagingDeep LearningMultimodal ImagingHumansGray MatterAdultNeuroimagingFemaleMaleFunctional NeuroimagingBrainYoung Adult

Stichwörter der Autor:innen

data fusionneuroimagingvision transformer

Autor:innen und Institutionen

  • Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia Tech, Emory, Atlanta, Georgia, USA.

ORCID-Profile: Yuda Bi, Anees Abrol

Förderung

  • National Science Foundation

Zitieren

APA

Bi, Y., Abrol, A., Fu, Z., & Calhoun, V. D. (2024). A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data. Human Brain Mapping, 45(17). https://doi.org/10.1002/hbm.26783

Vancouver

Bi Y, Abrol A, Fu Z, Calhoun VD. A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data. Human Brain Mapping. 2024;45(17). doi:10.1002/hbm.26783

BibTeX
@article{Bi2024,
  author = {Bi, Yuda and Abrol, Anees and Fu, Zening and Calhoun, Vince D},
  title = {A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data},
  journal = {Human Brain Mapping},
  year = {2024},
  volume = {45},
  number = {17},
  doi = {10.1002/hbm.26783}
}
Kostenlos starten →