Machine Learning to Prioritize High-Severity Patient Safety Events for Institutional Investigation: Algorithm Development and Validation Study
JMIR Medical Informatics · 2026
DOI: 10.2196/101393 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift JMIR Medical Informatics, 2026. Verfasst von Hongbo Chen und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada. Hauptthemen laut MEDLINE: Patient Safety, Risk Management, Machine Learning, Medical Errors. Die Arbeit nennt 58 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Themen: Künstliche Intelligenz in der Pflege, Patientensicherheit
Abstract (Originalsprache)
Background
Patient safety events (PSEs) are preventable incidents that cause, or have the potential to cause, harm to patients during their medical journey. Although incident reporting systems capture large volumes of such events, only a small proportion undergo comprehensive investigation due to the resource-intensive nature of the review process. Patient safety specialists are tasked with triaging PSEs to prioritize high-severity cases that warrant timely institutional investigation; however, the rapidly increasing volume of events has made manual triage increasingly impractical.
Objective
This study aimed to develop and validate ranking-based machine learning models for prioritizing high-severity PSEs and to compare their performance with conventional classification-based models and the reporter-assigned severity baseline.
Methods
A total of 101,239 PSE reports were retrospectively extracted from a large Canadian academic health system between January 2017 and March 2025. Each report included a free-text incident description and a corresponding severity label denoting the level of severity, assigned through an institutional review process. Both feature-based and transformer-based models were developed using ranking and classification frameworks. Model performance was evaluated using ranking metrics, including average precision and normalized discounted cumulative gain (NDCG) at multiple cutoff thresholds (k=10, 20, 50, 100, 200), as well as Precision@k and Recall@k. The best-performing model was further benchmarked against the reporter-assigned severity baseline.
Results
The ranking model, LLAMA3.1-RA, achieved the highest mean average precision of 0.94 (95% CI 0.90-0.98), representing a 22.1% relative improvement over the best classification model (LLAMA3.1-CL: mean 0.77, 95% CI 0.72-0.82; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada.
- Cerebras Systems, Toronto, ON, Canada.
- Health Quality Programs, Queen's University, Kingston, ON, Canada.
- Quality & Safety Department, University Health Network, Toronto, ON, Canada.
- Lawerence Bloomberg School of Nursing, University of Toronto, Toronto, ON, Canada.
- Division of Population Medicine, Cardiff University, Cardiff, Wales, United Kingdom.
ORCID-Profile: Hongbo Chen, Shehnaz Islam, Laura D Pozzobon, Lucas B Chartier, Eldan Cohen
Zitieren
APA
Chen, H., Islam, S., Pozzobon, L. D., Chartier, L. B., & Cohen, E. (2026). Machine Learning to Prioritize High-Severity Patient Safety Events for Institutional Investigation: Algorithm Development and Validation Study. JMIR Medical Informatics. https://doi.org/10.2196/101393
Vancouver
Chen H, Islam S, Pozzobon LD, Chartier LB, Cohen E. Machine Learning to Prioritize High-Severity Patient Safety Events for Institutional Investigation: Algorithm Development and Validation Study. JMIR Medical Informatics. 2026. doi:10.2196/101393
BibTeX
@article{Chen2026,
author = {Chen, Hongbo and Islam, Shehnaz and Pozzobon, Laura D and Chartier, Lucas B and Cohen, Eldan},
title = {Machine Learning to Prioritize High-Severity Patient Safety Events for Institutional Investigation: Algorithm Development and Validation Study},
journal = {JMIR Medical Informatics},
year = {2026},
doi = {10.2196/101393}
}
EUVIMED