EUVIMEDOpdag EUVIMED
KohortenstudieOpen access

Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study

Azade Tabaie iD, Alberta Tran iD, Tony Calabria iD, Sonita S Bennett iD, Arianna Milicia iD, William Weintraub iD, William James Gallagher iD, John Yosaitis iD, Laura C Schubel iD, Mary A Hill iD, Kelly Michelle Smith iD, Kristen Miller iD

Journal of Medical Internet Research · 2024-08-26 · 26 · e50935

DOI: 10.2196/50935 ↗

At a glance

Kohortenstudie in Journal of Medical Internet Research, published online on 2024-08-26. Written by Azade Tabaie and 11 co-authors. Institution: Center for Biostatistics, Informatics, and Data Science, MedStar Health Research Institute, Washington, DC, United States. Major MEDLINE topics: Natural Language Processing, Diagnostic Errors. Cited 15 times so far (Crossref); 43 references. The full text is freely available in PubMed Central. Funded by: AHRQ HHS.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Diagnostic errors are an underappreciated cause of preventable mortality in hospitals and pose a risk for severe patient harm and increase hospital length of stay.

Objective

This study aims to explore the potential of machine learning and natural language processing techniques in improving diagnostic safety surveillance. We conducted a rigorous evaluation of the feasibility and potential to use electronic health records clinical notes and existing case review data.

Methods

Safety Learning System case review data from 1 large health system composed of 10 hospitals in the mid-Atlantic region of the United States from February 2016 to September 2021 were analyzed. The case review outcome included opportunities for improvement including diagnostic opportunities for improvement. To supplement case review data, electronic health record clinical notes were extracted and analyzed. A simple logistic regression model along with 3 forms of logistic regression models (ie, Least Absolute Shrinkage and Selection Operator, Ridge, and Elastic Net) with regularization functions was trained on this data to compare classification performances in classifying patients who experienced diagnostic errors during hospitalization. Further, statistical tests were conducted to find significant differences between female and male patients who experienced diagnostic errors.

Results

In total, 126 (7.4%) patients (of 1704) had been identified by case reviewers as having experienced at least 1 diagnostic error. Patients who had experienced diagnostic error were grouped by sex: 59 (7.1%) of the 830 women and 67 (7.7%) of the 874 men. Among the patients who experienced a diagnostic error, female patients were older (median 72, IQR 66-80 vs median 67, IQR 57-76; P=.02), had higher rates of being admitted through general or internal medicine (69.5% vs 47.8%; P=.01), lower rates of cardiovascular-related admitted diagnosis (11.9% vs 28.4%; P=.02), and lower rates of being admitted through neurology department (2.3% vs 13.4%; P=.04). The Ridge model achieved the highest area under the receiver operating characteristic curve (0.885), specificity (0.797), positive predictive value (PPV; 0.24), and F1-score (0.369) in classifying patients who were at higher risk of diagnostic errors among hospitalized patients.

Conclusions

Our findings demonstrate that natural language processing can be a potential solution to more effectively identifying and selecting potential diagnostic error cases for review and therefore reducing the case review burden.

Subjects

MeSH terms (MEDLINE)

Natural Language ProcessingDiagnostic ErrorsHumansRetrospective StudiesMaleFemalePatient SafetyMiddle AgedElectronic Health RecordsMachine LearningAgedCohort StudiesUnited States

Author keywords

EHRNLPdiagnosticdiagnostic errorelectronic health recordshospitallength of staymachine learningmortalitynatural language processingpatient harmrisk

Authors and institutions

  • Center for Biostatistics, Informatics, and Data Science, MedStar Health Research Institute, Washington, DC, United States.
  • Department of Emergency Medicine, Georgetown University School of Medicine, Washington, DC, United States.
  • Department of Quality and Safety, MedStar Health Research Institute, Washington, DC, United States.
  • National Center for Human Factors in Healthcare, MedStar Health Research Institute, Washington, DC, United States.
  • Population Health, MedStar Health Research Institute, Washington, DC, United States.
  • Georgetown University School of Medicine, Washington, DC, United States.

ORCID profiles: Azade Tabaie, Alberta Tran, Tony Calabria, Sonita S Bennett, Arianna Milicia, William Weintraub, William James Gallagher, John Yosaitis, Laura C Schubel, Mary A Hill, Kelly Michelle Smith, Kristen Miller

Funding

  • AHRQ HHS

Cite

APA

Tabaie, A., Tran, A., Calabria, T., Bennett, S. S., Milicia, A., Weintraub, W., Gallagher, W. J., Yosaitis, J., Schubel, L. C., Hill, M. A., Smith, K. M., & Miller, K. (2024). Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study. Journal of Medical Internet Research, 26, e50935. https://doi.org/10.2196/50935

Vancouver

Tabaie A, Tran A, Calabria T, Bennett SS, Milicia A, Weintraub W, et al. Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study. Journal of Medical Internet Research. 2024;26:e50935. doi:10.2196/50935

BibTeX
@article{Tabaie2024,
  author = {Tabaie, Azade and Tran, Alberta and Calabria, Tony and Bennett, Sonita S and Milicia, Arianna and Weintraub, William and Gallagher, William James and Yosaitis, John and Schubel, Laura C and Hill, Mary A and Smith, Kelly Michelle and Miller, Kristen},
  title = {Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e50935},
  doi = {10.2196/50935}
}
Start for free →