EUVIMEDObjevte EUVIMED
ValidierungsstudieOpen access

Machine Learning to Prioritize High-Severity Patient Safety Events for Institutional Investigation: Algorithm Development and Validation Study

Hongbo Chen iD, Shehnaz Islam iD, Laura D Pozzobon iD, Lucas B Chartier iD, Eldan Cohen iD

JMIR Medical Informatics · 2026

DOI: 10.2196/101393 ↗

At a glance

Validierungsstudie in JMIR Medical Informatics, 2026. Written by Hongbo Chen and 4 co-authors. Institution: Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada. Major MEDLINE topics: Patient Safety, Risk Management, Machine Learning, Medical Errors. 58 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Patient safety events (PSEs) are preventable incidents that cause, or have the potential to cause, harm to patients during their medical journey. Although incident reporting systems capture large volumes of such events, only a small proportion undergo comprehensive investigation due to the resource-intensive nature of the review process. Patient safety specialists are tasked with triaging PSEs to prioritize high-severity cases that warrant timely institutional investigation; however, the rapidly increasing volume of events has made manual triage increasingly impractical.

Objective

This study aimed to develop and validate ranking-based machine learning models for prioritizing high-severity PSEs and to compare their performance with conventional classification-based models and the reporter-assigned severity baseline.

Methods

A total of 101,239 PSE reports were retrospectively extracted from a large Canadian academic health system between January 2017 and March 2025. Each report included a free-text incident description and a corresponding severity label denoting the level of severity, assigned through an institutional review process. Both feature-based and transformer-based models were developed using ranking and classification frameworks. Model performance was evaluated using ranking metrics, including average precision and normalized discounted cumulative gain (NDCG) at multiple cutoff thresholds (k=10, 20, 50, 100, 200), as well as Precision@k and Recall@k. The best-performing model was further benchmarked against the reporter-assigned severity baseline.

Results

The ranking model, LLAMA3.1-RA, achieved the highest mean average precision of 0.94 (95% CI 0.90-0.98), representing a 22.1% relative improvement over the best classification model (LLAMA3.1-CL: mean 0.77, 95% CI 0.72-0.82; P

Subjects

MeSH terms (MEDLINE)

Patient SafetyRisk ManagementMachine LearningMedical ErrorsSecondary Data AnalysisHumansReference ValuesRetrospective StudiesCanadaTriageClassification Algorithms

Author keywords

AI in health careclinical decision supportincident reporting systemslearning-to-rankmachine learningnatural language processingpatient safety eventsseverity triage

Authors and institutions

  • Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada.
  • Cerebras Systems, Toronto, ON, Canada.
  • Health Quality Programs, Queen's University, Kingston, ON, Canada.
  • Quality & Safety Department, University Health Network, Toronto, ON, Canada.
  • Lawerence Bloomberg School of Nursing, University of Toronto, Toronto, ON, Canada.
  • Division of Population Medicine, Cardiff University, Cardiff, Wales, United Kingdom.

ORCID profiles: Hongbo Chen, Shehnaz Islam, Laura D Pozzobon, Lucas B Chartier, Eldan Cohen

Cite

APA

Chen, H., Islam, S., Pozzobon, L. D., Chartier, L. B., & Cohen, E. (2026). Machine Learning to Prioritize High-Severity Patient Safety Events for Institutional Investigation: Algorithm Development and Validation Study. JMIR Medical Informatics. https://doi.org/10.2196/101393

Vancouver

Chen H, Islam S, Pozzobon LD, Chartier LB, Cohen E. Machine Learning to Prioritize High-Severity Patient Safety Events for Institutional Investigation: Algorithm Development and Validation Study. JMIR Medical Informatics. 2026. doi:10.2196/101393

BibTeX
@article{Chen2026,
  author = {Chen, Hongbo and Islam, Shehnaz and Pozzobon, Laura D and Chartier, Lucas B and Cohen, Eldan},
  title = {Machine Learning to Prioritize High-Severity Patient Safety Events for Institutional Investigation: Algorithm Development and Validation Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  doi = {10.2196/101393}
}
Start for free →