Measuring the Impact of AI in the Diagnosis of Hospitalized Patients
JAMA · 19.12.2023 · 330 (23) · 2275
DOI: 10.1001/jama.2023.22295 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JAMA, erschienen am 19.12.2023. Verfasst von Sarah Jabbour und 7 weiteren Autor:innen. Beteiligte Einrichtung: Computer Science and Engineering, University of Michigan, Ann Arbor. Hauptthemen laut MEDLINE: Artificial Intelligence, Respiratory Insufficiency, Clinical Competence. Die Arbeit wurde bisher 254-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NHLBI NIH HHS.
Abstract (Originalsprache)
Importance Artificial intelligence (AI) could support clinicians when diagnosing hospitalized patients; however, systematic bias in AI models could worsen clinician diagnostic accuracy. Recent regulatory guidance has called for AI models to include explanations to mitigate errors made by models, but the effectiveness of this strategy has not been established.
Objectives
To evaluate the impact of systematically biased AI on clinician diagnostic accuracy and to determine if image-based AI model explanations can mitigate model errors. Design, Setting, and Participants Randomized clinical vignette survey study administered between April 2022 and January 2023 across 13 US states involving hospitalist physicians, nurse practitioners, and physician assistants.
Interventions
Clinicians were shown 9 clinical vignettes of patients hospitalized with acute respiratory failure, including their presenting symptoms, physical examination, laboratory results, and chest radiographs. Clinicians were then asked to determine the likelihood of pneumonia, heart failure, or chronic obstructive pulmonary disease as the underlying cause(s) of each patient’s acute respiratory failure. To establish baseline diagnostic accuracy, clinicians were shown 2 vignettes without AI model input. Clinicians were then randomized to see 6 vignettes with AI model input with or without AI model explanations. Among these 6 vignettes, 3 vignettes included standard-model predictions, and 3 vignettes included systematically biased model predictions. Main Outcomes and Measures Clinician diagnostic accuracy for pneumonia, heart failure, and chronic obstructive pulmonary disease.
Results
Median participant age was 34 years (IQR, 31-39) and 241 (57.7%) were female. Four hundred fifty-seven clinicians were randomized and completed at least 1 vignette, with 231 randomized to AI model predictions without explanations, and 226 randomized to AI model predictions with explanations. Clinicians’ baseline diagnostic accuracy was 73.0% (95% CI, 68.3% to 77.8%) for the 3 diagnoses. When shown a standard AI model without explanations, clinician accuracy increased over baseline by 2.9 percentage points (95% CI, 0.5 to 5.2) and by 4.4 percentage points (95% CI, 2.0 to 6.9) when clinicians were also shown AI model explanations. Systematically biased AI model predictions decreased clinician accuracy by 11.3 percentage points (95% CI, 7.2 to 15.5) compared with baseline and providing biased AI model predictions with explanations decreased clinician accuracy by 9.1 percentage points (95% CI, 4.9 to 13.2) compared with baseline, representing a nonsignificant improvement of 2.3 percentage points (95% CI, −2.7 to 7.2) compared with the systematically biased AI model. Conclusions and Relevance Although standard AI models improve diagnostic accuracy, systematically biased AI models reduced diagnostic accuracy, and commonly used image-based AI model explanations did not mitigate this harmful effect. Trial Registration ClinicalTrials.gov Identifier: NCT06098950
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Computer Science and Engineering, University of Michigan, Ann Arbor.
- Now with Computer Science Courant Institute, New York University, New York.
- Now with Electrical and Computer Engineering Tandon School of Engineering, New York University, New York.
- Pulmonary and Critical Care Medicine, Department of Internal Medicine, University of Michigan Medical School, Ann Arbor.
- Department of Radiology, University of Michigan Medical School, Ann Arbor.
Förderung
- NHLBI NIH HHS
Zitieren
APA
Jabbour, S., Fouhey, D., Shepard, S., Valley, T. S., Kazerooni, E. A., Banovic, N., Wiens, J., & Sjoding, M. W. (2023). Measuring the Impact of AI in the Diagnosis of Hospitalized Patients. JAMA, 330(23), 2275. https://doi.org/10.1001/jama.2023.22295
Vancouver
Jabbour S, Fouhey D, Shepard S, Valley TS, Kazerooni EA, Banovic N, et al. Measuring the Impact of AI in the Diagnosis of Hospitalized Patients. JAMA. 2023;330(23):2275. doi:10.1001/jama.2023.22295
BibTeX
@article{Jabbour2023,
author = {Jabbour, Sarah and Fouhey, David and Shepard, Stephanie and Valley, Thomas S and Kazerooni, Ella A and Banovic, Nikola and Wiens, Jenna and Sjoding, Michael W},
title = {Measuring the Impact of AI in the Diagnosis of Hospitalized Patients},
journal = {JAMA},
year = {2023},
volume = {330},
number = {23},
pages = {2275},
doi = {10.1001/jama.2023.22295}
}
EUVIMED