EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

The future of AI clinicians: assessing the modern standard of chatbots and their approach to diagnostic uncertainty

Ryan S Huang, Ali Benour, Joel Kemppainen, Fok-Han Leung

BMC Medical Education · 2024

DOI: 10.1186/s12909-024-06115-5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift BMC Medical Education, 2024. Verfasst von Ryan S Huang und 3 weiteren Autor:innen. Beteiligte Einrichtung: Temerty Faculty of Medicine, University of Toronto, Health Centre at 80 Bond, St. Michael's Hospital, 80 Bond Street, Toronto, ON, M5B1X2, Canada. ry…. Hauptthemen laut MEDLINE: Artificial Intelligence, Internship and Residency, Clinical Competence. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence (AI) chatbots have demonstrated proficiency in structured knowledge assessments; however, there is limited research on their performance in scenarios involving diagnostic uncertainty, which requires careful interpretation and complex decision-making. This study aims to evaluate the efficacy of AI chatbots, GPT-4o and Claude-3, in addressing medical scenarios characterized by diagnostic uncertainty relative to Family Medicine residents.

Methods

Questions with diagnostic uncertainty were extracted from the Progress Tests administered by the Department of Family and Community Medicine at the University of Toronto between 2022 and 2023. Diagnostic uncertainty questions were defined as those presenting clinical scenarios where symptoms, clinical findings, and patient histories do not converge on a definitive diagnosis, necessitating nuanced diagnostic reasoning and differential diagnosis. These questions were administered to a cohort of 320 Family Medicine residents in their first (PGY-1) and second (PGY-2) postgraduate years and inputted into GPT-4o and Claude-3. Errors were categorized into statistical, information, and logical errors. Statistical analyses were conducted using a binomial generalized estimating equation model, paired t-tests, and chi-squared tests.

Results

Compared to the residents, both chatbots scored lower on diagnostic uncertainty questions ( p < 0.01). PGY-1 residents achieved a correctness rate of 61.1% (95% CI: 58.4–63.7), and PGY-2 residents achieved 63.3% (95% CI: 60.7–66.1). In contrast, Claude-3 correctly answered 57.7% ( n = 52/90) of questions, and GPT-4o correctly answered 53.3% ( n = 48/90). Claude-3 had a longer mean response time (24.0 s, 95% CI: 21.0-32.5 vs. 12.4 s, 95% CI: 9.3–15.3; p < 0.01) and produced longer answers (2001 characters, 95% CI: 1845–2212 vs. 1596 characters, 95% CI: 1395–1705; p < 0.01) compared to GPT-4o. Most errors by GPT-4o were logical errors (62.5%).

Conclusions

While AI chatbots like GPT-4o and Claude-3 demonstrate potential in handling structured medical knowledge, their performance in scenarios involving diagnostic uncertainty remains suboptimal compared to human residents.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceInternship and ResidencyClinical CompetenceHumansUncertaintyFamily PracticeEducational Measurement

Stichwörter der Autor:innen

Artificial intelligenceDecision makingDiagnostic uncertainty

Autor:innen und Institutionen

  • Temerty Faculty of Medicine, University of Toronto, Health Centre at 80 Bond, St. Michael's Hospital, 80 Bond Street, Toronto, ON, M5B1X2, Canada. ry.huang@mail.utoronto.ca.
  • Department of Medicine, Royal College of Surgeons in Ireland, Dublin, Leinster, Ireland.
  • Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada.

Zitieren

APA

Huang, R. S., Benour, A., Kemppainen, J., & Leung, F. H. (2024). The future of AI clinicians: assessing the modern standard of chatbots and their approach to diagnostic uncertainty. BMC Medical Education. https://doi.org/10.1186/s12909-024-06115-5

Vancouver

Huang RS, Benour A, Kemppainen J, Leung FH. The future of AI clinicians: assessing the modern standard of chatbots and their approach to diagnostic uncertainty. BMC Medical Education. 2024. doi:10.1186/s12909-024-06115-5

BibTeX
@article{Huang2024,
  author = {Huang, Ryan S and Benour, Ali and Kemppainen, Joel and Leung, Fok-Han},
  title = {The future of AI clinicians: assessing the modern standard of chatbots and their approach to diagnostic uncertainty},
  journal = {BMC Medical Education},
  year = {2024},
  doi = {10.1186/s12909-024-06115-5}
}
Kostenlos starten →