Accuracy and Reliability of Chatbot Responses to Physician Questions
JAMA Network Open · 02.10.2023 · 6 (10) · e2336483
DOI: 10.1001/jamanetworkopen.2023.36483 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMA Network Open, online erschienen am 02.10.2023. Verfasst von Rachel S Goodman und 34 weiteren Autor:innen. Beteiligte Einrichtung: Vanderbilt University School of Medicine, Nashville, Tennessee. Hauptthemen laut MEDLINE: Artificial Intelligence, Physicians. Die Arbeit wurde bisher 496-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCI NIH HHS, CSRD VA, NHLBI NIH HHS.
Abstract (Originalsprache)
Importance Natural language processing tools, such as ChatGPT (generative pretrained transformer, hereafter referred to as chatbot), have the potential to radically enhance the accessibility of medical information for health professionals and patients. Assessing the safety and efficacy of these tools in answering physician-generated questions is critical to determining their suitability in clinical settings, facilitating complex decision-making, and optimizing health care efficiency.
Objective
To assess the accuracy and comprehensiveness of chatbot-generated responses to physician-developed medical queries, highlighting the reliability and limitations of artificial intelligence–generated medical information. Design, Setting, and Participants Thirty-three physicians across 17 specialties generated 284 medical questions that they subjectively classified as easy, medium, or hard with either binary (yes or no) or descriptive answers. The physicians then graded the chatbot-generated answers to these questions for accuracy (6-point Likert scale with 1 being completely incorrect and 6 being completely correct) and completeness (3-point Likert scale, with 1 being incomplete and 3 being complete plus additional context). Scores were summarized with descriptive statistics and compared using the Mann-Whitney U test or the Kruskal-Wallis test. The study (including data analysis) was conducted from January to May 2023. Main Outcomes and Measures Accuracy, completeness, and consistency over time and between 2 different versions (GPT-3.5 and GPT-4) of chatbot-generated medical responses.
Results
Across all questions (n = 284) generated by 33 physicians (31 faculty members and 2 recent graduates from residency or fellowship programs) across 17 specialties, the median accuracy score was 5.5 (IQR, 4.0-6.0) (between almost completely and complete correct) with a mean (SD) score of 4.8 (1.6) (between mostly and almost completely correct). The median completeness score was 3.0 (IQR, 2.0-3.0) (complete and comprehensive) with a mean (SD) score of 2.5 (0.7). For questions rated easy, medium, and hard, the median accuracy scores were 6.0 (IQR, 5.0-6.0), 5.5 (IQR, 5.0-6.0), and 5.0 (IQR, 4.0-6.0), respectively (mean [SD] scores were 5.0 [1.5], 4.7 [1.7], and 4.6 [1.6], respectively; P = .05). Accuracy scores for binary and descriptive questions were similar (median score, 6.0 [IQR, 4.0-6.0] vs 5.0 [IQR, 3.4-6.0]; mean [SD] score, 4.9 [1.6] vs 4.7 [1.6]; P = .07). Of 36 questions with scores of 1.0 to 2.0, 34 were requeried or regraded 8 to 17 days later with substantial improvement (median score 2.0 [IQR, 1.0-3.0] vs 4.0 [IQR, 2.0-5.3]; P < .01). A subset of questions, regardless of initial scores (version 3.5), were regenerated and rescored using version 4 with improvement (mean accuracy [SD] score, 5.2 [1.5] vs 5.7 [0.8]; median score, 6.0 [IQR, 5.0-6.0] for original and 6.0 [IQR, 6.0-6.0] for rescored; P = .002). Conclusions and Relevance In this cross-sectional study, chatbot generated largely accurate information to diverse medical queries as judged by academic physician specialists with improvement over time, although it had important limitations. Further research and model development are needed to correct inaccuracies and for validation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Vanderbilt University School of Medicine, Nashville, Tennessee.
- Department of Dermatology, Vanderbilt University Medical Center, Nashville, Tennessee.
- Department of Allergy, Pulmonology, and Critical Care, Vanderbilt University Medical Center, Nashville, Tennessee.
- Department of Neurology, Vanderbilt University Medical Center, Nashville, Tennessee.
- Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee.
- Department of Urology, Vanderbilt University Medical Center, Nashville, Tennessee.
Förderung
- NCI NIH HHS
- CSRD VA
- NHLBI NIH HHS
- NIDDK NIH HHS
Zitieren
APA
Goodman, R. S., Patrinely, J. R., Stone, C. A., Zimmerman, E., Donald, R. R., Chang, S. S., Berkowitz, S. T., Finn, A. P., Jahangir, E., Scoville, E. A., Reese, T. S., Friedman, D. L., Bastarache, J. A., van der Heijden, Y. F., Wright, J. J., Ye, F., Carter, N., Alexander, M. R., Choe, J. H., …, & Johnson, D. B. (2023). Accuracy and Reliability of Chatbot Responses to Physician Questions. JAMA Network Open, 6(10), e2336483. https://doi.org/10.1001/jamanetworkopen.2023.36483
Vancouver
Goodman RS, Patrinely JR, Stone CA, Zimmerman E, Donald RR, Chang SS, et al. Accuracy and Reliability of Chatbot Responses to Physician Questions. JAMA Network Open. 2023;6(10):e2336483. doi:10.1001/jamanetworkopen.2023.36483
BibTeX
@article{Goodman2023,
author = {Goodman, Rachel S and Patrinely, J Randall and Stone, Cosby A and Zimmerman, Eli and Donald, Rebecca R and Chang, Sam S and Berkowitz, Sean T and Finn, Avni P and Jahangir, Eiman and Scoville, Elizabeth A and Reese, Tyler S and Friedman, Debra L and Bastarache, Julie A and van der Heijden, Yuri F and Wright, Jordan J and Ye, Fei and Carter, Nicholas and Alexander, Matthew R and Choe, Jennifer H and Chastain, Cody A and Zic, John A and Horst, Sara N and Turker, Isik and Agarwal, Rajiv and Osmundson, Evan and Idrees, Kamran and Kiernan, Colleen M and Padmanabhan, Chandrasekhar and Bailey, Christina E and Schlegel, Cameron E and Chambless, Lola B and Gibson, Michael K and Osterman, Travis J and Wheless, Lee E and Johnson, Douglas B},
title = {Accuracy and Reliability of Chatbot Responses to Physician Questions},
journal = {JAMA Network Open},
year = {2023},
volume = {6},
number = {10},
pages = {e2336483},
doi = {10.1001/jamanetworkopen.2023.36483}
}
EUVIMED