Talking to the Brain: Using Large Language Models as Proxies to Model Brain Semantic Features
Human Brain Mapping · 2026
DOI: 10.1002/hbm.70588 ↗Kurz gefasst
Fachartikel in der Zeitschrift Human Brain Mapping, 2026. Verfasst von Xin Liu und 2 weiteren Autor:innen. Beteiligte Einrichtung: Philosophy and Social Science Laboratory of Reading and Development in Children and Adolescents (South China Normal University), Ministry of Educatio…. Hauptthemen laut MEDLINE: Semantics, Brain Mapping, Brain, Models, Neurological. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Research Center for Brain Cognition and Human Development, Key-Area Research and Development Program of Guangdong Province, Striving for the First-Class, Improving Weak Links and Highlighting Features (SIH) Key Discipline for Psychology in South China Normal University.
Abstract (Originalsprache)
Utilizing naturalistic stimuli while maintaining experimental rigor presents a significant challenge in neuroimaging research, primarily due to the complexities of annotation and high‐dimensional analysis. To address this, we introduce a novel paradigm that leverages multimodal large language models (LLMs) as high‐throughput, automated annotation tools to extract human‐understandable semantic features. This “talking to the brain” approach employs a Visual Question Answering (VQA) strategy to transform complex visual scenes into structured semantic feature vectors, which are then used to predict voxel‐wise brain activity. As a critical methodological validation, our model successfully replicated established neural correlates for categories such as face and building. Expanding beyond these benchmarks, we mapped the cortical activation patterns of 80 diverse semantic labels and constructed a data‐driven brain semantic similarity space. This space revealed robust clusters reflecting both functional categories and real‐world contextual associations. This innovative methodology offers a scalable solution for investigating brain semantic organization, overcoming the limitations of manual annotation and paving the way for more ecologically valid explorations of human cognition.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Philosophy and Social Science Laboratory of Reading and Development in Children and Adolescents (South China Normal University), Ministry of Education Center for Studies of Psychological Application, South China Normal…
- Center for Studies of Psychological Application, South China Normal University, Guangzhou, China.
- Key Laboratory of Brain, Cognition and Education Sciences (South China Normal University), Ministry of Education, Guangzhou, China.
- School of Psychology, South China Normal University, Guangzhou, China.
- Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou, China.
ORCID-Profile: Ziyue Zhang, Jingxin Nie
Förderung
- Research Center for Brain Cognition and Human Development
- Key-Area Research and Development Program of Guangdong Province
- Striving for the First-Class, Improving Weak Links and Highlighting Features (SIH) Key Discipline for Psychology in South China Normal University
Zitieren
APA
Liu, X., Zhang, Z., & Nie, J. (2026). Talking to the Brain: Using Large Language Models as Proxies to Model Brain Semantic Features. Human Brain Mapping. https://doi.org/10.1002/hbm.70588
Vancouver
Liu X, Zhang Z, Nie J. Talking to the Brain: Using Large Language Models as Proxies to Model Brain Semantic Features. Human Brain Mapping. 2026. doi:10.1002/hbm.70588
BibTeX
@article{Liu2026,
author = {Liu, Xin and Zhang, Ziyue and Nie, Jingxin},
title = {Talking to the Brain: Using Large Language Models as Proxies to Model Brain Semantic Features},
journal = {Human Brain Mapping},
year = {2026},
doi = {10.1002/hbm.70588}
}
EUVIMED