Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study
Journal of Medical Internet Research · 2024-08-26 · 26 · e50935
DOI: 10.2196/50935 ↗At a glance
Kohortenstudie in Journal of Medical Internet Research, published online on 2024-08-26. Written by Azade Tabaie and 11 co-authors. Institution: Center for Biostatistics, Informatics, and Data Science, MedStar Health Research Institute, Washington, DC, United States. Major MEDLINE topics: Natural Language Processing, Diagnostic Errors. Cited 15 times so far (Crossref); 43 references. The full text is freely available in PubMed Central. Funded by: AHRQ HHS.
Abstract (original language)
Background
Diagnostic errors are an underappreciated cause of preventable mortality in hospitals and pose a risk for severe patient harm and increase hospital length of stay.
Objective
This study aims to explore the potential of machine learning and natural language processing techniques in improving diagnostic safety surveillance. We conducted a rigorous evaluation of the feasibility and potential to use electronic health records clinical notes and existing case review data.
Methods
Safety Learning System case review data from 1 large health system composed of 10 hospitals in the mid-Atlantic region of the United States from February 2016 to September 2021 were analyzed. The case review outcome included opportunities for improvement including diagnostic opportunities for improvement. To supplement case review data, electronic health record clinical notes were extracted and analyzed. A simple logistic regression model along with 3 forms of logistic regression models (ie, Least Absolute Shrinkage and Selection Operator, Ridge, and Elastic Net) with regularization functions was trained on this data to compare classification performances in classifying patients who experienced diagnostic errors during hospitalization. Further, statistical tests were conducted to find significant differences between female and male patients who experienced diagnostic errors.
Results
In total, 126 (7.4%) patients (of 1704) had been identified by case reviewers as having experienced at least 1 diagnostic error. Patients who had experienced diagnostic error were grouped by sex: 59 (7.1%) of the 830 women and 67 (7.7%) of the 874 men. Among the patients who experienced a diagnostic error, female patients were older (median 72, IQR 66-80 vs median 67, IQR 57-76; P=.02), had higher rates of being admitted through general or internal medicine (69.5% vs 47.8%; P=.01), lower rates of cardiovascular-related admitted diagnosis (11.9% vs 28.4%; P=.02), and lower rates of being admitted through neurology department (2.3% vs 13.4%; P=.04). The Ridge model achieved the highest area under the receiver operating characteristic curve (0.885), specificity (0.797), positive predictive value (PPV; 0.24), and F1-score (0.369) in classifying patients who were at higher risk of diagnostic errors among hospitalized patients.
Conclusions
Our findings demonstrate that natural language processing can be a potential solution to more effectively identifying and selecting potential diagnostic error cases for review and therefore reducing the case review burden.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Center for Biostatistics, Informatics, and Data Science, MedStar Health Research Institute, Washington, DC, United States.
- Department of Emergency Medicine, Georgetown University School of Medicine, Washington, DC, United States.
- Department of Quality and Safety, MedStar Health Research Institute, Washington, DC, United States.
- National Center for Human Factors in Healthcare, MedStar Health Research Institute, Washington, DC, United States.
- Population Health, MedStar Health Research Institute, Washington, DC, United States.
- Georgetown University School of Medicine, Washington, DC, United States.
ORCID profiles: Azade Tabaie, Alberta Tran, Tony Calabria, Sonita S Bennett, Arianna Milicia, William Weintraub, William James Gallagher, John Yosaitis, Laura C Schubel, Mary A Hill, Kelly Michelle Smith, Kristen Miller
Funding
- AHRQ HHS
Cite
APA
Tabaie, A., Tran, A., Calabria, T., Bennett, S. S., Milicia, A., Weintraub, W., Gallagher, W. J., Yosaitis, J., Schubel, L. C., Hill, M. A., Smith, K. M., & Miller, K. (2024). Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study. Journal of Medical Internet Research, 26, e50935. https://doi.org/10.2196/50935
Vancouver
Tabaie A, Tran A, Calabria T, Bennett SS, Milicia A, Weintraub W, et al. Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study. Journal of Medical Internet Research. 2024;26:e50935. doi:10.2196/50935
BibTeX
@article{Tabaie2024,
author = {Tabaie, Azade and Tran, Alberta and Calabria, Tony and Bennett, Sonita S and Milicia, Arianna and Weintraub, William and Gallagher, William James and Yosaitis, John and Schubel, Laura C and Hill, Mary A and Smith, Kelly Michelle and Miller, Kristen},
title = {Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e50935},
doi = {10.2196/50935}
}
EUVIMED