The future of AI clinicians: assessing the modern standard of chatbots and their approach to diagnostic uncertainty
BMC Medical Education · 2024
DOI: 10.1186/s12909-024-06115-5 ↗Kurz gefasst
Fachartikel in der Zeitschrift BMC Medical Education, 2024. Verfasst von Ryan S Huang und 3 weiteren Autor:innen. Beteiligte Einrichtung: Temerty Faculty of Medicine, University of Toronto, Health Centre at 80 Bond, St. Michael's Hospital, 80 Bond Street, Toronto, ON, M5B1X2, Canada. ry…. Hauptthemen laut MEDLINE: Artificial Intelligence, Internship and Residency, Clinical Competence. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Artificial intelligence (AI) chatbots have demonstrated proficiency in structured knowledge assessments; however, there is limited research on their performance in scenarios involving diagnostic uncertainty, which requires careful interpretation and complex decision-making. This study aims to evaluate the efficacy of AI chatbots, GPT-4o and Claude-3, in addressing medical scenarios characterized by diagnostic uncertainty relative to Family Medicine residents.
Methods
Questions with diagnostic uncertainty were extracted from the Progress Tests administered by the Department of Family and Community Medicine at the University of Toronto between 2022 and 2023. Diagnostic uncertainty questions were defined as those presenting clinical scenarios where symptoms, clinical findings, and patient histories do not converge on a definitive diagnosis, necessitating nuanced diagnostic reasoning and differential diagnosis. These questions were administered to a cohort of 320 Family Medicine residents in their first (PGY-1) and second (PGY-2) postgraduate years and inputted into GPT-4o and Claude-3. Errors were categorized into statistical, information, and logical errors. Statistical analyses were conducted using a binomial generalized estimating equation model, paired t-tests, and chi-squared tests.
Results
Compared to the residents, both chatbots scored lower on diagnostic uncertainty questions ( p < 0.01). PGY-1 residents achieved a correctness rate of 61.1% (95% CI: 58.4–63.7), and PGY-2 residents achieved 63.3% (95% CI: 60.7–66.1). In contrast, Claude-3 correctly answered 57.7% ( n = 52/90) of questions, and GPT-4o correctly answered 53.3% ( n = 48/90). Claude-3 had a longer mean response time (24.0 s, 95% CI: 21.0-32.5 vs. 12.4 s, 95% CI: 9.3–15.3; p < 0.01) and produced longer answers (2001 characters, 95% CI: 1845–2212 vs. 1596 characters, 95% CI: 1395–1705; p < 0.01) compared to GPT-4o. Most errors by GPT-4o were logical errors (62.5%).
Conclusions
While AI chatbots like GPT-4o and Claude-3 demonstrate potential in handling structured medical knowledge, their performance in scenarios involving diagnostic uncertainty remains suboptimal compared to human residents.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Temerty Faculty of Medicine, University of Toronto, Health Centre at 80 Bond, St. Michael's Hospital, 80 Bond Street, Toronto, ON, M5B1X2, Canada. ry.huang@mail.utoronto.ca.
- Department of Medicine, Royal College of Surgeons in Ireland, Dublin, Leinster, Ireland.
- Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada.
Zitieren
APA
Huang, R. S., Benour, A., Kemppainen, J., & Leung, F. H. (2024). The future of AI clinicians: assessing the modern standard of chatbots and their approach to diagnostic uncertainty. BMC Medical Education. https://doi.org/10.1186/s12909-024-06115-5
Vancouver
Huang RS, Benour A, Kemppainen J, Leung FH. The future of AI clinicians: assessing the modern standard of chatbots and their approach to diagnostic uncertainty. BMC Medical Education. 2024. doi:10.1186/s12909-024-06115-5
BibTeX
@article{Huang2024,
author = {Huang, Ryan S and Benour, Ali and Kemppainen, Joel and Leung, Fok-Han},
title = {The future of AI clinicians: assessing the modern standard of chatbots and their approach to diagnostic uncertainty},
journal = {BMC Medical Education},
year = {2024},
doi = {10.1186/s12909-024-06115-5}
}
EUVIMED