EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Evaluating Large Language Models for Analysing Safety Risks in Healthcare Incident Reports

Kerstin Denecke, Helmut Paula

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti250867 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Kerstin Denecke und 1 weiteren Autor:innen. Beteiligte Einrichtung: Bern University of Applied Sciences, Bern, Switzerland. Hauptthemen laut MEDLINE: Natural Language Processing, Risk Management, Medical Errors, Patient Safety. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Incident reports provide a rich source for analysing safety risks in healthcare systems. To support the timely analysis and interpretation of incident reports, natural language processing (NLP) can be applied. The aim of this paper is to evaluate the potential of large language models (LLMs) in extracting the causes of incidents and identifying contributing factors from incident reports. As dataset, we considered 10,063 messages from CIRRNET®, the Swiss national database for critical incidents in healthcare. We applied the LLM Gemma-2 to extract events, causes and contributing factors and group them along themes. 100 event reports were assessed manually regarding quality of extraction. Events were extracted with 92% accuracy, causes with 84% and contributing factors with 72% accuracy. Extraction of contributing factors fails as the LLM hallucinates or interprets. We conclude that LLMs show potential in analysing incident reports and can improve the efficiency and consistency of incident analysis.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingRisk ManagementMedical ErrorsPatient SafetyData MiningSafety ManagementSwitzerlandHumansLarge Language Models

Stichwörter der Autor:innen

Critical incident reporting systemLarge language modelNatural language processingPatient safety event reportingText analysis

Autor:innen und Institutionen

  • Bern University of Applied Sciences, Bern, Switzerland.
  • Stiftung Patientensicherheit Schweiz, Zürich, Switzerland.

Zitieren

APA

Denecke, K., & Paula, H. (2025). Evaluating Large Language Models for Analysing Safety Risks in Healthcare Incident Reports. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250867

Vancouver

Denecke K, Paula H. Evaluating Large Language Models for Analysing Safety Risks in Healthcare Incident Reports. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250867

BibTeX
@article{Denecke2025,
  author = {Denecke, Kerstin and Paula, Helmut},
  title = {Evaluating Large Language Models for Analysing Safety Risks in Healthcare Incident Reports},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti250867}
}
Kostenlos starten →