Fine-tuning and evaluating large language models for patient safety tasks: classification of contributing factors in incident reports
Journal of the American Medical Informatics Association · 01.08.2026 · 33 (8) · 1494-1503
DOI: 10.1093/jamia/ocag068 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 20.05.2026. Verfasst von Ying Wang und 5 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia. Hauptthemen laut MEDLINE: Large Language Models, Patient Safety, Risk Management, Medical Errors. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Australian National Health and Medical Research Council (NHMRC).
Themen: Patientensicherheit
Abstract (Originalsprache)
Objective
To evaluate and compare the performance of large language models (LLMs) in identifying contributing factors (CFs) underlying patient safety incident investigations. Materials and Methods Four open-source, lightweight LLMs, including BERT, LLaMA2, GPT2, and Phi-2 were applied to classify CFs across 6 sociotechnical system-levels encompassing 12 categories (eg, person, task, and organizational factors). Reports of real-world patient safety investigations from public health systems were extracted and labelled by domain experts (n_report/CFs = 300/1338). Data were split into training (n = 852), validation (n = 98), and test sets (n = 388). Performance was evaluated using specificity, precision, recall, and F1 scores.
Results
The fine-tuned encoder-based BERT model achieved the highest performance, with a micro-averaged F1 score of 63.6%, outperforming all decoder-based models. Among the decoder models, Phi-2 demonstrated the strongest performance (F1 = 54.9%), exceeding both LLaMA2 and GPT2. BERT performed consistently across 6 system-levels but often misclassified “organization” as “person”.
Discussion
LLMs hold promise for automating the extraction of CFs from complex safety narratives, particularly for frequently reported system-levels such as “person” and “tasks”. Such automation may substantially reduce the manual effort required to analyse reports of patient safety investigations while supporting more consistent analysis across large incident datasets.
Conclusion
Applying LLMs to analyse the underlying causes of patient safety incidents depends on developing high-quality, domain-specific datasets that enhance the representation of patient safety knowledge and improve model understanding of incident causation. Improving data coverage for rare system-levels is essential to address the current limitations of LLMs in capturing nuanced patient safety concepts and domain-specific reasoning.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia.
- Centre for Healthcare Resilience and Implementation Science, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia.
ORCID-Profile: Ying Wang, Peter Hibbert, Farah Magrabi
Förderung
- Australian National Health and Medical Research Council (NHMRC)
Zitieren
APA
Wang, Y., Bowditch, L., Molloy, C., Yu, Y., Hibbert, P., & Magrabi, F. (2026). Fine-tuning and evaluating large language models for patient safety tasks: classification of contributing factors in incident reports. Journal of the American Medical Informatics Association, 33(8), 1494-1503. https://doi.org/10.1093/jamia/ocag068
Vancouver
Wang Y, Bowditch L, Molloy C, Yu Y, Hibbert P, Magrabi F. Fine-tuning and evaluating large language models for patient safety tasks: classification of contributing factors in incident reports. Journal of the American Medical Informatics Association. 2026;33(8):1494-1503. doi:10.1093/jamia/ocag068
BibTeX
@article{Wang2026,
author = {Wang, Ying and Bowditch, Lorelle and Molloy, Charlotte and Yu, Yinghua and Hibbert, Peter and Magrabi, Farah},
title = {Fine-tuning and evaluating large language models for patient safety tasks: classification of contributing factors in incident reports},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
volume = {33},
number = {8},
pages = {1494--1503},
doi = {10.1093/jamia/ocag068}
}
EUVIMED