Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image
JMIR Aging · 21.09.2026 · 9 · e93279-e93279
DOI: 10.2196/93279 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Aging, online erschienen am 21.09.2026. Verfasst von Felix Agbavor und 1 weiteren Autor:innen. Beteiligte Einrichtung: School of Biomedical Engineering and Science, Drexel University, Philadelphia, PA, United States. Hauptthemen laut MEDLINE: Attention, Alzheimer Disease, Dementia. Die Arbeit nennt 36 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIA NIH HHS.
Themen: Demenz
Abstract (Originalsprache)
Background
Alzheimer disease (AD) is a leading cause of dementia, and there is growing interest in scalable approaches for early screening using speech-based tasks. While prior work has demonstrated promising results using either transcript-based language features or acoustic cues, most approaches remain unimodal or rely on simple fusion strategies that do not explicitly consider interactions across modalities.
Objective
In this study, we propose an attention-based trimodal fusion framework that integrates text, audio, and image representations of the Cookie Theft picture, which serves as the shared visual stimulus in the picture-description task.
Methods
Our method uses a new bidirectional cross-attention mechanism to achieve a unified multimodal embedding for downstream tasks. We evaluate the approach on 2 tasks: AD detection by classifying whether the participant has AD or not, and AD severity assessment by predicting Mini-Mental Status Examination cognitive scores.
Results
On the AD detection task, trimodal fusion achieves the best overall performance ( F 1 -score=0.8667, area under the receiver operating characteristic curve=0.9032), outperforming unimodal baselines, bimodal fusion, and conventional early or late fusion methods. For AD severity assessment, the proposed multimodal representation reduces prediction error of root mean squared error to about 4.20, improving over both unimodal and bimodal fusion settings. We further perform the ablation analysis to show that bidirectional cross-attention consistently outperforms conventional unidirectional cross-attention.
Conclusions
These results demonstrate that attention-based multimodal fusion can enhance dementia prediction from picture-description responses and provide a strong foundation for developing multimodal cognitive screening pipelines.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Biomedical Engineering and Science, Drexel University, Philadelphia, PA, United States.
- Division of Artificial Intelligence and the Humanities, The Hong Kong Polytechnic University, HHB717, 7/F, 8 Hung Lok Road, Hung Hom, Kowloon, China (Hong Kong), 852 2766-7697.
- Department of Language Science and Technology, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong).
- Departments of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong).
Förderung
- NIA NIH HHS
Zitieren
APA
Agbavor, F., & Liang, H. (2026). Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image. JMIR Aging, 9, e93279-e93279. https://doi.org/10.2196/93279
Vancouver
Agbavor F, Liang H. Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image. JMIR Aging. 2026;9:e93279-e93279. doi:10.2196/93279
BibTeX
@article{Agbavor2026,
author = {Agbavor, Felix and Liang, Hualou},
title = {Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image},
journal = {JMIR Aging},
year = {2026},
volume = {9},
pages = {e93279--e93279},
doi = {10.2196/93279}
}
EUVIMED