EUVIMEDEUVIMED entdecken
Crossref

Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli

Panpan Chen iD, Chi Zhang, Bao Li iD, Li Tong, Shuxiao Ma, Linyuan Wang iD, Long Cao, Ziya Yu, Bin Yan iD

Journal of Neural Engineering · 2025

DOI: 10.1088/1741-2552/ae1bd9 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Neural Engineering, 2025. Verfasst von Panpan Chen und 8 weiteren Autor:innen. Beteiligte Einrichtung: Henan Key Laboratory of Imaging and Intelligent Processing, Information Engineering University, Zhengzhou 450000, People's Republic of China. Hauptthemen laut MEDLINE: Magnetic Resonance Imaging, Photic Stimulation, Visual Perception, Neural Networks, Computer. Die Arbeit nennt 49 Literaturangaben. Gefördert von: National Natural Science Foundation of China, STI2030-Major Projects, Central Plains Science and Technology Innovation Leading Talent Project.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective . Modeling neural encoding of visual stimuli often uses deep neural networks (DNNs) to predict human brain response to external stimuli. However, each DNN depends on networks tailored for computer vision tasks, resulting in suboptimal brain correspondence. On the other hand, when end-to-end optimizing the encoding process for specific brain regions, challenges like training difficulties arise. Additionally, these models mostly focus on visual information processing, while the human brain integrates multi-modal information such as language to achieve a comprehensive understanding. Approach . To address these limitations, this paper proposes a multi-modal prompt learning (PL) model for neural encoding of dynamic natural stimuli. Specifically, we leverage the powerful representation ability of pre-trained foundation models and fine-tune them using our multi-modal prompts. These prompts, which include textual and visual prompts tailored to each specific regions of interest, can adapt foundation models to neural encoding tasks with fewer trainable parameters. We use the CLIP For video Clip retrieval (CLIP4clip) and Video Masked Autoencoder V2 (videoMAEv2) for feature extraction with backbone freezing, refine the representations via PL, and map the fused multi-modal features to predict voxel-wise brain responses. Main results . Extensive experiments on two functional magnetic resonance imaging video datasets demonstrate that our method outperforms existing fine-tuning methods and public models. Significance . This work highlights the potential of prompt-based fine-tuning strategies in bridging the gap between foundation models and neural encoding tasks.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Magnetic Resonance ImagingPhotic StimulationVisual PerceptionNeural Networks, ComputerBrainDeep LearningHumansMaleFemaleAdult

Stichwörter der Autor:innen

dynamic natural stimulifMRI visual encodingfoundation modelsmulti-modal prompt learningvoxel-wise mapping

Autor:innen und Institutionen

  • Henan Key Laboratory of Imaging and Intelligent Processing, Information Engineering University, Zhengzhou 450000, People's Republic of China.

ORCID-Profile: Panpan Chen, Bao Li, Linyuan Wang, Bin Yan

Förderung

  • National Natural Science Foundation of China
  • STI2030-Major Projects
  • Central Plains Science and Technology Innovation Leading Talent Project

Zitieren

APA

Chen, P., Zhang, C., Li, B., Tong, L., Ma, S., Wang, L., Cao, L., Yu, Z., & Yan, B. (2025). Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli. Journal of Neural Engineering. https://doi.org/10.1088/1741-2552/ae1bd9

Vancouver

Chen P, Zhang C, Li B, Tong L, Ma S, Wang L, et al. Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli. Journal of Neural Engineering. 2025. doi:10.1088/1741-2552/ae1bd9

BibTeX
@article{Chen2025,
  author = {Chen, Panpan and Zhang, Chi and Li, Bao and Tong, Li and Ma, Shuxiao and Wang, Linyuan and Cao, Long and Yu, Ziya and Yan, Bin},
  title = {Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli},
  journal = {Journal of Neural Engineering},
  year = {2025},
  doi = {10.1088/1741-2552/ae1bd9}
}
Kostenlos starten →