Monitoring performance of clinical artificial intelligence in health care: a scoping review
Monitoring der klinischen Leistung von künstlicher Intelligenz im Gesundheitswesen: Scoping-Review
JBI Evidence Synthesis · 2024-12 · 22 (12) · 2423-2446
DOI: 10.11124/jbies-24-00042 ↗Auf einen Blick
Der Review beschreibt Methoden zum Monitoring klinischer KI-Systeme. Es zeigte sich eine geringe Evidenz- und Leitlinienlage für die praktische Umsetzung.
Fragestellung
Welche Methoden zum Monitoring der klinischen Leistung von KI-Systemen im Gesundheitswesen werden beschrieben, getestet oder implementiert?
Methode
Scoping-Review mit systematischen Datenbanksuchen in MEDLINE, Embase, Scopus und ProQuest Dissertations and Theses Global sowie Zitations- und Grauliteratursuche.
Ergebnisse
39 Evidenzquellen wurden eingeschlossen. Am häufigsten waren Meinungsartikel/narrative Reviews (33 %) und Simulationsstudien (33 %); eine Leitlinie lieferte nur begrenzte Orientierung.
Einschränkungen
Nur eine Leitlinie; wenig konkrete Anleitung zu spezifischen Metriken und statistischen Methoden.
Schlussfolgerung der Autor:innen
Der Review fasst Methoden zum Monitoring von KI im Gesundheitswesen zusammen und zeigt einen Mangel an Evidenz und praktischer Anleitung; weiterer Forschungs- und Diskussionsbedarf besteht.
Kernaussagen
- 39 Quellen zu Monitoring-Methoden für klinische KI
- Am häufigsten: AUROC, Sensitivität, Spezifität und prädiktive Werte
- Wenig spezifische Evidenz und Leitlinien für die Praxis
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Scoping Review in der Zeitschrift JBI Evidence Synthesis, 2024-12. Verfasst von Eline Sandvig Andersen und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biochemistry and Immunology, Lillebaelt Hospital - University Hospital of Southern Denmark, Vejle, Denmark. Hauptthemen laut MEDLINE: Artificial Intelligence. Die Arbeit wurde bisher 71-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Objective
The objective of this review was to provide an overview of the diverse methods described, tested, or implemented for monitoring performance of clinical artificial intelligence (AI) systems, while also summarizing the arguments given for or against these methods.
Introduction
The integration of AI in clinical decision-making is steadily growing. Performances of AI systems evolve over time, necessitating ongoing performance monitoring. However, the evidence on specific monitoring methods is sparse and heterogeneous. Thus, an overview of the evidence on this topic is warranted to guide further research on clinical AI monitoring. Inclusion criteria: We included publications detailing metrics or statistical processes employed in systematic, continuous, or repeated initiatives aimed at evaluating or predicting the clinical performance of AI models with direct implications for patient management in health care. No limitations on language or publication date were enforced.
Methods
We performed systematic database searches in MEDLINE (Ovid), Embase (Ovid), Scopus, and ProQuest Dissertations and Theses Global, supplemented by backward and forward citation searches and gray literature searches. Two or more independent reviewers conducted title and abstract screening, full-text evaluation, and data extraction using a tool developed by the authors. During extraction, the methods identified were divided into subcategories. The results are presented narratively and summarized in tables and graphs.
Results
Thirty-nine sources of evidence were included in the review, with the most abundant source types being opinion papers/narrative reviews (33%) and simulation studies (33%). One guideline on the topic was identified, offering limited guidance on specific metrics and statistical methods. The number of sources included increased year by year, with almost 4 times as many sources included in 2023 compared with 2019. The most commonly reported performance metrics were traditional metrics from the medical literature, including area under the receiver operating characteristics curve (AUROC), sensitivity, specificity, and predictive values, although few arguments were given supporting these choices. Some studies reported on metrics and statistical processing specifically designed to monitor clinical AI.
Conclusion
This review provides a summary of the methods described for monitoring AI in health care. It reveals a relative scarcity of evidence and guidance for specific practical implementation of performance monitoring of clinical AI. This underscores the imperative for further research, discussion, and guidance regarding the specifics of implementing monitoring for clinical AI. The steady increase in the number of relevant sources published per year suggests that this area of research is gaining increased focus, and the amount of evidence and guidance available will likely increase significantly over the coming years. Review registration: Open Science Framework https://osf.io/afkrn
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Biochemistry and Immunology, Lillebaelt Hospital - University Hospital of Southern Denmark, Vejle, Denmark.
- Department of Regional Health Research, University of Southern Denmark, Lillebælt Hospital (Kolding and Vejle), Denmark.
- Department of Clinical Biochemistry, Odense University Hospital, Odense, Denmark.
- Department of Mathematics and Computer Science, University of Southern Denmark, Odense, Denmark.
- Department of Thoracic Surgery, the First Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, China.
ORCID-Profile: Eline Sandvig Andersen, Johan Baden Birk-Korch, Rasmus Søgaard Hansen, Line Haugaard Fly, Richard Röttger, Claus Lohman Brasen, Ivan Brandslund, Jonna Skov Madsen
Zitieren
APA
Andersen, E. S., Birk-Korch, J. B., Hansen, R. S., Fly, L. H., Röttger, R., Arcani, D. M. C., Brasen, C. L., Brandslund, I., & Madsen, J. S. (2024). Monitoring performance of clinical artificial intelligence in health care: a scoping review. JBI Evidence Synthesis, 22(12), 2423-2446. https://doi.org/10.11124/jbies-24-00042
Vancouver
Andersen ES, Birk-Korch JB, Hansen RS, Fly LH, Röttger R, Arcani DMC, et al. Monitoring performance of clinical artificial intelligence in health care: a scoping review. JBI Evidence Synthesis. 2024;22(12):2423-2446. doi:10.11124/jbies-24-00042
BibTeX
@article{Andersen2024,
author = {Andersen, Eline Sandvig and Birk-Korch, Johan Baden and Hansen, Rasmus Søgaard and Fly, Line Haugaard and Röttger, Richard and Arcani, Diana Maria Cespedes and Brasen, Claus Lohman and Brandslund, Ivan and Madsen, Jonna Skov},
title = {Monitoring performance of clinical artificial intelligence in health care: a scoping review},
journal = {JBI Evidence Synthesis},
year = {2024},
volume = {22},
number = {12},
pages = {2423--2446},
doi = {10.11124/jbies-24-00042}
}
EUVIMED