Evaluating Large Language Models for Analysing Safety Risks in Healthcare Incident Reports
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti250867 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Kerstin Denecke und 1 weiteren Autor:innen. Beteiligte Einrichtung: Bern University of Applied Sciences, Bern, Switzerland. Hauptthemen laut MEDLINE: Natural Language Processing, Risk Management, Medical Errors, Patient Safety. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Incident reports provide a rich source for analysing safety risks in healthcare systems. To support the timely analysis and interpretation of incident reports, natural language processing (NLP) can be applied. The aim of this paper is to evaluate the potential of large language models (LLMs) in extracting the causes of incidents and identifying contributing factors from incident reports. As dataset, we considered 10,063 messages from CIRRNET®, the Swiss national database for critical incidents in healthcare. We applied the LLM Gemma-2 to extract events, causes and contributing factors and group them along themes. 100 event reports were assessed manually regarding quality of extraction. Events were extracted with 92% accuracy, causes with 84% and contributing factors with 72% accuracy. Extraction of contributing factors fails as the LLM hallucinates or interprets. We conclude that LLMs show potential in analysing incident reports and can improve the efficiency and consistency of incident analysis.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Bern University of Applied Sciences, Bern, Switzerland.
- Stiftung Patientensicherheit Schweiz, Zürich, Switzerland.
Zitieren
APA
Denecke, K., & Paula, H. (2025). Evaluating Large Language Models for Analysing Safety Risks in Healthcare Incident Reports. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250867
Vancouver
Denecke K, Paula H. Evaluating Large Language Models for Analysing Safety Risks in Healthcare Incident Reports. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250867
BibTeX
@article{Denecke2025,
author = {Denecke, Kerstin and Paula, Helmut},
title = {Evaluating Large Language Models for Analysing Safety Risks in Healthcare Incident Reports},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti250867}
}
EUVIMED