EUVIMEDEUVIMED entdecken
Open Access

Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study

Hamish Fraser iD, Daven Crossland iD, Ian Bacher iD, Megan Ranney iD, Tracy Madsen iD, Ross Hilliard iD

JMIR mHealth and uHealth · 2023

DOI: 10.2196/49995 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR mHealth and uHealth, 2023. Verfasst von Hamish Fraser und 5 weiteren Autor:innen. Beteiligte Einrichtung: Brown Center for Biomedical Informatics, The Warren Alpert Medical School of Brown University, Providence, RI, United States. Hauptthemen laut MEDLINE: Triage, Physicians. Die Arbeit wurde bisher 133-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NHLBI NIH HHS, NIGMS NIH HHS, AHRQ HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Diagnosis is a core component of effective health care, but misdiagnosis is common and can put patients at risk. Diagnostic decision support systems can play a role in improving diagnosis by physicians and other health care workers. Symptom checkers (SCs) have been designed to improve diagnosis and triage (ie, which level of care to seek) by patients.

Objective

The aim of this study was to evaluate the performance of the new large language model ChatGPT (versions 3.5 and 4.0), the widely used WebMD SC, and an SC developed by Ada Health in the diagnosis and triage of patients with urgent or emergent clinical problems compared with the final emergency department (ED) diagnoses and physician reviews.

Methods

We used previously collected, deidentified, self-report data from 40 patients presenting to an ED for care who used the Ada SC to record their symptoms prior to seeing the ED physician. Deidentified data were entered into ChatGPT versions 3.5 and 4.0 and WebMD by a research assistant blinded to diagnoses and triage. Diagnoses from all 4 systems were compared with the previously abstracted final diagnoses in the ED as well as with diagnoses and triage recommendations from three independent board-certified ED physicians who had blindly reviewed the self-report clinical data from Ada. Diagnostic accuracy was calculated as the proportion of the diagnoses from ChatGPT, Ada SC, WebMD SC, and the independent physicians that matched at least one ED diagnosis (stratified as top 1 or top 3). Triage accuracy was calculated as the number of recommendations from ChatGPT, WebMD, or Ada that agreed with at least 2 of the independent physicians or were rated “unsafe” or “too cautious.” Results Overall, 30 and 37 cases had sufficient data for diagnostic and triage analysis, respectively. The rate of top-1 diagnosis matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 9 (30%), 12 (40%), 10 (33%), and 12 (40%), respectively, with a mean rate of 47% for the physicians. The rate of top-3 diagnostic matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 19 (63%), 19 (63%), 15 (50%), and 17 (57%), respectively, with a mean rate of 69% for physicians. The distribution of triage results for Ada was 62% (n=23) agree, 14% unsafe (n=5), and 24% (n=9) too cautious; that for ChatGPT 3.5 was 59% (n=22) agree, 41% (n=15) unsafe, and 0% (n=0) too cautious; that for ChatGPT 4.0 was 76% (n=28) agree, 22% (n=8) unsafe, and 3% (n=1) too cautious; and that for WebMD was 70% (n=26) agree, 19% (n=7) unsafe, and 11% (n=4) too cautious. The unsafe triage rate for ChatGPT 3.5 (41%) was significantly higher (P=.009) than that of Ada (14%).

Conclusions

ChatGPT 3.5 had high diagnostic accuracy but a high unsafe triage rate. ChatGPT 4.0 had the poorest diagnostic accuracy, but a lower unsafe triage rate and the highest triage agreement with the physicians. The Ada and WebMD SCs performed better overall than ChatGPT. Unsupervised patient use of ChatGPT for diagnosis and triage is not recommended without improvements to triage accuracy and extensive clinical evaluation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

TriagePhysiciansHumansEmergency Service, HospitalHealth PersonnelSelf Report

Stichwörter der Autor:innen

ChatGPTChatGPT-3.5ChatGPT-4.0LLMaccuracyappapplicationdiagnosediagnosisemergencyemergency patientlanguage modelmachine learningself-diagnoseself-diagnosis

Autor:innen und Institutionen

  • Brown Center for Biomedical Informatics, The Warren Alpert Medical School of Brown University, Providence, RI, United States.
  • Department of Health Services, Policy and Practice, Brown University School of Public Health, Providence, RI, United States.
  • Department of Epidemiology, Brown University School of Public Health, Providence, RI, United States.
  • School of Public Health, Yale University, New Haven, CT, United States.
  • Department of Emergency Medicine, The Warren Alpert Medical School of Brown University, Providence, RI, United States.
  • Department of Internal Medicine, Maine Medical Center, Portland, ME, United States.

ORCID-Profile: Hamish Fraser, Daven Crossland, Ian Bacher, Megan Ranney, Tracy Madsen, Ross Hilliard

Förderung

  • NHLBI NIH HHS
  • NIGMS NIH HHS
  • AHRQ HHS

Zitieren

APA

Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023). Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study. JMIR mHealth and uHealth. https://doi.org/10.2196/49995

Vancouver

Fraser H, Crossland D, Bacher I, Ranney M, Madsen T, Hilliard R. Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study. JMIR mHealth and uHealth. 2023. doi:10.2196/49995

BibTeX
@article{Fraser2023,
  author = {Fraser, Hamish and Crossland, Daven and Bacher, Ian and Ranney, Megan and Madsen, Tracy and Hilliard, Ross},
  title = {Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study},
  journal = {JMIR mHealth and uHealth},
  year = {2023},
  doi = {10.2196/49995}
}
Kostenlos starten →