EUVIMEDOpdag EUVIMED
Open access

Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image

Multimodale Demenzvorhersage mit Large Language Models: Cross-Attention über Text, Audio und Bild

Felix Agbavor, Hualou Liang

JMIR Aging · 2026-09-21 · 9 · e93279-e93279

DOI: 10.2196/93279 ↗

Auf einen Blick

Eine trimodale, auf Attention basierende Fusion aus Text, Audio und Bild erzielte bei der AD-Erkennung die beste Leistung und senkte bei der Schweregradschätzung den Fehler.

Fragestellung

Verbessert eine attentionbasierte trimodale Fusion aus Text, Audio und Bild die Erkennung von Alzheimer-Demenz und die Schweregradschätzung anhand von Bildbeschreibungen?

Methode

Modell mit bidirektionaler Cross-Attention für eine multimodale Einbettung; Evaluation auf 2 Aufgaben: AD-Erkennung und Schweregradschätzung.

Ergebnisse

Bei der AD-Erkennung erreichte die trimodale Fusion F1=0,8667 und AUC=0,9032 und war besser als unimodale, bimodale sowie frühe/späte Fusionsansätze. Für die Schweregradschätzung lag der RMSE bei etwa 4,20.

Schlussfolgerung der Autor:innen

Die Ergebnisse zeigen, dass attentionbasierte multimodale Fusion die Demenzvorhersage aus Bildbeschreibungen verbessern kann und eine Grundlage für multimodale kognitive Screening-Pipelines bietet.

Kernaussagen

  • Trimodale Fusion mit bester AD-Erkennungsleistung
  • F1=0,8667; AUC=0,9032
  • Schweregradschätzung mit RMSE von etwa 4,20

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Journal article in JMIR Aging, published online on 2026-09-21. Written by Felix Agbavor and 1 co-authors. Institution: School of Biomedical Engineering and Science, Drexel University, Philadelphia, PA, United States. Major MEDLINE topics: Attention, Alzheimer Disease, Dementia. 36 references. The full text is freely available in PubMed Central. Funded by: NIA NIH HHS.

Auf einen BlickAbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Alzheimer disease (AD) is a leading cause of dementia, and there is growing interest in scalable approaches for early screening using speech-based tasks. While prior work has demonstrated promising results using either transcript-based language features or acoustic cues, most approaches remain unimodal or rely on simple fusion strategies that do not explicitly consider interactions across modalities.

Objective

In this study, we propose an attention-based trimodal fusion framework that integrates text, audio, and image representations of the Cookie Theft picture, which serves as the shared visual stimulus in the picture-description task.

Methods

Our method uses a new bidirectional cross-attention mechanism to achieve a unified multimodal embedding for downstream tasks. We evaluate the approach on 2 tasks: AD detection by classifying whether the participant has AD or not, and AD severity assessment by predicting Mini-Mental Status Examination cognitive scores.

Results

On the AD detection task, trimodal fusion achieves the best overall performance ( F 1 -score=0.8667, area under the receiver operating characteristic curve=0.9032), outperforming unimodal baselines, bimodal fusion, and conventional early or late fusion methods. For AD severity assessment, the proposed multimodal representation reduces prediction error of root mean squared error to about 4.20, improving over both unimodal and bimodal fusion settings. We further perform the ablation analysis to show that bidirectional cross-attention consistently outperforms conventional unidirectional cross-attention.

Conclusions

These results demonstrate that attention-based multimodal fusion can enhance dementia prediction from picture-description responses and provide a strong foundation for developing multimodal cognitive screening pipelines.

Subjects

MeSH terms (MEDLINE)

AttentionAlzheimer DiseaseDementiaHumansLarge Language ModelsFemaleAged, 80 and over

Author keywords

Alzheimer diseasecognitive assessmentcross-attentionmultimodal fusionspeech analysis

Authors and institutions

  • School of Biomedical Engineering and Science, Drexel University, Philadelphia, PA, United States.
  • Division of Artificial Intelligence and the Humanities, The Hong Kong Polytechnic University, HHB717, 7/F, 8 Hung Lok Road, Hung Hom, Kowloon, China (Hong Kong), 852 2766-7697.
  • Department of Language Science and Technology, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong).
  • Departments of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong).

Funding

  • NIA NIH HHS

Cite

APA

Agbavor, F., & Liang, H. (2026). Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image. JMIR Aging, 9, e93279-e93279. https://doi.org/10.2196/93279

Vancouver

Agbavor F, Liang H. Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image. JMIR Aging. 2026;9:e93279-e93279. doi:10.2196/93279

BibTeX
@article{Agbavor2026,
  author = {Agbavor, Felix and Liang, Hualou},
  title = {Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image},
  journal = {JMIR Aging},
  year = {2026},
  volume = {9},
  pages = {e93279--e93279},
  doi = {10.2196/93279}
}
Start for free →