Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli
Journal of Neural Engineering · 2025
DOI: 10.1088/1741-2552/ae1bd9 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Neural Engineering, 2025. Verfasst von Panpan Chen und 8 weiteren Autor:innen. Beteiligte Einrichtung: Henan Key Laboratory of Imaging and Intelligent Processing, Information Engineering University, Zhengzhou 450000, People's Republic of China. Hauptthemen laut MEDLINE: Magnetic Resonance Imaging, Photic Stimulation, Visual Perception, Neural Networks, Computer. Die Arbeit nennt 49 Literaturangaben. Gefördert von: National Natural Science Foundation of China, STI2030-Major Projects, Central Plains Science and Technology Innovation Leading Talent Project.
Abstract (Originalsprache)
Objective . Modeling neural encoding of visual stimuli often uses deep neural networks (DNNs) to predict human brain response to external stimuli. However, each DNN depends on networks tailored for computer vision tasks, resulting in suboptimal brain correspondence. On the other hand, when end-to-end optimizing the encoding process for specific brain regions, challenges like training difficulties arise. Additionally, these models mostly focus on visual information processing, while the human brain integrates multi-modal information such as language to achieve a comprehensive understanding. Approach . To address these limitations, this paper proposes a multi-modal prompt learning (PL) model for neural encoding of dynamic natural stimuli. Specifically, we leverage the powerful representation ability of pre-trained foundation models and fine-tune them using our multi-modal prompts. These prompts, which include textual and visual prompts tailored to each specific regions of interest, can adapt foundation models to neural encoding tasks with fewer trainable parameters. We use the CLIP For video Clip retrieval (CLIP4clip) and Video Masked Autoencoder V2 (videoMAEv2) for feature extraction with backbone freezing, refine the representations via PL, and map the fused multi-modal features to predict voxel-wise brain responses. Main results . Extensive experiments on two functional magnetic resonance imaging video datasets demonstrate that our method outperforms existing fine-tuning methods and public models. Significance . This work highlights the potential of prompt-based fine-tuning strategies in bridging the gap between foundation models and neural encoding tasks.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Henan Key Laboratory of Imaging and Intelligent Processing, Information Engineering University, Zhengzhou 450000, People's Republic of China.
ORCID-Profile: Panpan Chen, Bao Li, Linyuan Wang, Bin Yan
Förderung
- National Natural Science Foundation of China
- STI2030-Major Projects
- Central Plains Science and Technology Innovation Leading Talent Project
Zitieren
APA
Chen, P., Zhang, C., Li, B., Tong, L., Ma, S., Wang, L., Cao, L., Yu, Z., & Yan, B. (2025). Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli. Journal of Neural Engineering. https://doi.org/10.1088/1741-2552/ae1bd9
Vancouver
Chen P, Zhang C, Li B, Tong L, Ma S, Wang L, et al. Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli. Journal of Neural Engineering. 2025. doi:10.1088/1741-2552/ae1bd9
BibTeX
@article{Chen2025,
author = {Chen, Panpan and Zhang, Chi and Li, Bao and Tong, Li and Ma, Shuxiao and Wang, Linyuan and Cao, Long and Yu, Ziya and Yan, Bin},
title = {Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli},
journal = {Journal of Neural Engineering},
year = {2025},
doi = {10.1088/1741-2552/ae1bd9}
}
EUVIMED