Performance Assessment of ChatGPT versus Bard in Detecting Alzheimer’s Dementia
Diagnostics · 15.04.2024 · 14 (8) · 817
DOI: 10.3390/diagnostics14080817 ↗Kurz gefasst
Fachartikel in der Zeitschrift Diagnostics, online erschienen am 15.04.2024. Verfasst von Balamurali B T und 1 weiteren Autor:innen. Beteiligte Einrichtung: Science, Mathematics & Technology (SMT), Singapore University of Technology & Design, 8 Somapah Rd, Singapore 487372, Singapore. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) find increasing applications in many fields. Here, three LLM chatbots (ChatGPT-3.5, ChatGPT-4, and Bard) are assessed in their current form, as publicly available, for their ability to recognize Alzheimer’s dementia (AD) and Cognitively Normal (CN) individuals using textual input derived from spontaneous speech recordings. A zero-shot learning approach is used at two levels of independent queries, with the second query (chain-of-thought prompting) eliciting more detailed information than the first. Each LLM chatbot’s performance is evaluated on the prediction generated in terms of accuracy, sensitivity, specificity, precision, and F1 score. LLM chatbots generated a three-class outcome (“AD”, “CN”, or “Unsure”). When positively identifying AD, Bard produced the highest true-positives (89% recall) and highest F1 score (71%), but tended to misidentify CN as AD, with high confidence (low “Unsure” rates); for positively identifying CN, GPT-4 resulted in the highest true-negatives at 56% and highest F1 score (62%), adopting a diplomatic stance (moderate “Unsure” rates). Overall, the three LLM chatbots can identify AD vs. CN, surpassing chance-levels, but do not currently satisfy the requirements for clinical application.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Science, Mathematics & Technology (SMT), Singapore University of Technology & Design, 8 Somapah Rd, Singapore 487372, Singapore.
ORCID-Profile: Balamurali B T, Jer-Ming Chen
Zitieren
APA
B T, B., & Chen, J. M. (2024). Performance Assessment of ChatGPT versus Bard in Detecting Alzheimer’s Dementia. Diagnostics, 14(8), 817. https://doi.org/10.3390/diagnostics14080817
Vancouver
B T B, Chen JM. Performance Assessment of ChatGPT versus Bard in Detecting Alzheimer’s Dementia. Diagnostics. 2024;14(8):817. doi:10.3390/diagnostics14080817
BibTeX
@article{BT2024,
author = {B T, Balamurali and Chen, Jer-Ming},
title = {Performance Assessment of ChatGPT versus Bard in Detecting Alzheimer’s Dementia},
journal = {Diagnostics},
year = {2024},
volume = {14},
number = {8},
pages = {817},
doi = {10.3390/diagnostics14080817}
}
EUVIMED