EUVIMEDEUVIMED entdecken
Open Access

A Machine Learning Approach with Human-AI Collaboration for Automated Classification of Patient Safety Event Reports: Algorithm Development and Validation Study

Ein Machine-Learning-Ansatz mit Human-AI-Kollaboration zur automatisierten Klassifikation von Patientensicherheitsereignis-Berichten: Entwicklungs- und Validierungsstudie

Hongbo Chen iD, Eldan Cohen iD, Dulaney Wilson iD, Myrtede Alfred iD

JMIR Human Factors · 25.01.2024 · 11 · e53378

DOI: 10.2196/53378 ↗

Auf einen Blick

ML-Modelle klassifizierten 861 Patientensicherheitsereignis-Berichte besser mit kontextueller als mit statischer Textrepräsentation. Ein Interface mit Erklärungen wurde für die Human-AI-Kollaboration entworfen.

Fragestellung

Wie gut lassen sich PSE-Berichte mit ML-Klassifikatoren auf Basis kontextueller Textrepräsentationen automatisch klassifizieren, und wie kann dies mit Erklärbarkeit unterstützt werden?

Methode

Entwicklungs- und Validierungsstudie mit 861 PSE-Berichten aus einer maternalen Station eines akademischen Krankenhauses im Südosten der USA; Vergleich statischer und kontextueller Textrepräsentationen; Bewertung mit Multiklassen-Metriken und Confusion Matrix; LIME zur Erklärbarkeit.

Ergebnisse

Der bestperformende Klassifikator mit kontextueller Repräsentation erreichte 75,4 % (95/126) Accuracy gegenüber 66,7 % (84/126) mit statischer Textrepräsentation. Ein Interface für PSE-Reporting wurde entworfen.

Einschränkungen

LIME zeigte, dass der Klassifikator gelegentlich auf arbiträre Wörter zurückgreift; daher ist menschliche Aufsicht erforderlich.

Schlussfolgerung der Autor:innen

Die Autor:innen schließen, dass kontextuelle Textrepräsentationen die Genauigkeit der PSE-Berichtsklassifikation deutlich verbessern und das Interface die Grundlage für Human-AI-Kollaboration legt.

Kernaussagen

  • 75,4 % Accuracy mit kontextueller Textrepräsentation
  • 66,7 % Accuracy mit statischer Textrepräsentation
  • LIME belegte gelegentlich die Nutzung arbiträrer Wörter

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Human Factors, online erschienen am 25.01.2024. Verfasst von Hongbo Chen und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Mechanical & Industrial Engineering, Faculty of Applied Science & Engineering, University of Toronto, Toronto, ON, Canada. Hauptthemen laut MEDLINE: Artificial Intelligence, Patient Safety, Oxides, Calcium Compounds. Die Arbeit wurde bisher 31-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: AHRQ HHS.

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Adverse events refer to incidents with potential or actual harm to patients in hospitals. These events are typically documented through patient safety event (PSE) reports, which consist of detailed narratives providing contextual information on the occurrences. Accurate classification of PSE reports is crucial for patient safety monitoring. However, this process faces challenges due to inconsistencies in classifications and the sheer volume of reports. Recent advancements in text representation, particularly contextual text representation derived from transformer-based language models, offer a promising solution for more precise PSE report classification. Integrating the machine learning (ML) classifier necessitates a balance between human expertise and artificial intelligence (AI). Central to this integration is the concept of explainability, which is crucial for building trust and ensuring effective human-AI collaboration.

Objective

This study aims to investigate the efficacy of ML classifiers trained using contextual text representation in automatically classifying PSE reports. Furthermore, the study presents an interface that integrates the ML classifier with the explainability technique to facilitate human-AI collaboration for PSE report classification.

Methods

This study used a data set of 861 PSE reports from a large academic hospital’s maternity units in the Southeastern United States. Various ML classifiers were trained with both static and contextual text representations of PSE reports. The trained ML classifiers were evaluated with multiclass classification metrics and the confusion matrix. The local interpretable model-agnostic explanations (LIME) technique was used to provide the rationale for the ML classifier’s predictions. An interface that integrates the ML classifier with the LIME technique was designed for incident reporting systems.

Results

The top-performing classifier using contextual representation was able to obtain an accuracy of 75.4% (95/126) compared to an accuracy of 66.7% (84/126) by the top-performing classifier trained using static text representation. A PSE reporting interface has been designed to facilitate human-AI collaboration in PSE report classification. In this design, the ML classifier recommends the top 2 most probable event types, along with the explanations for the prediction, enabling PSE reporters and patient safety analysts to choose the most suitable one. The LIME technique showed that the classifier occasionally relies on arbitrary words for classification, emphasizing the necessity of human oversight.

Conclusions

This study demonstrates that training ML classifiers with contextual text representations can significantly enhance the accuracy of PSE report classification. The interface designed in this study lays the foundation for human-AI collaboration in the classification of PSE reports. The insights gained from this research enhance the decision-making process in PSE report classification, enabling hospitals to more efficiently identify potential risks and hazards and enabling patient safety analysts to take timely actions to prevent patient harm.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligencePatient SafetyOxidesCalcium CompoundsFemalePregnancyHumansAlgorithmsMachine Learning

Stichwörter der Autor:innen

LIMEaccidentaccidentsartificial intelligenceblack boxclassificationclassifiercollaborationdesigndocumentdocumentationdocumentsexplainabilityexplainablehuman-AI

Substanzen

limeOxidesCalcium Compounds

Autor:innen und Institutionen

  • Department of Mechanical & Industrial Engineering, Faculty of Applied Science & Engineering, University of Toronto, Toronto, ON, Canada.
  • Department of Public Health Sciences, College of Medicine, Medical University of South Carolina, Charleston, SC, United States.

ORCID-Profile: Hongbo Chen, Eldan Cohen, Dulaney Wilson, Myrtede Alfred

Förderung

  • AHRQ HHS

Zitieren

APA

Chen, H., Cohen, E., Wilson, D., & Alfred, M. (2024). A Machine Learning Approach with Human-AI Collaboration for Automated Classification of Patient Safety Event Reports: Algorithm Development and Validation Study. JMIR Human Factors, 11, e53378. https://doi.org/10.2196/53378

Vancouver

Chen H, Cohen E, Wilson D, Alfred M. A Machine Learning Approach with Human-AI Collaboration for Automated Classification of Patient Safety Event Reports: Algorithm Development and Validation Study. JMIR Human Factors. 2024;11:e53378. doi:10.2196/53378

BibTeX
@article{Chen2024,
  author = {Chen, Hongbo and Cohen, Eldan and Wilson, Dulaney and Alfred, Myrtede},
  title = {A Machine Learning Approach with Human-AI Collaboration for Automated Classification of Patient Safety Event Reports: Algorithm Development and Validation Study},
  journal = {JMIR Human Factors},
  year = {2024},
  volume = {11},
  pages = {e53378},
  doi = {10.2196/53378}
}
Kostenlos starten →