EUVIMEDEUVIMED entdecken
Scoping ReviewOpen Access · CC BY-NC-ND 4.0

Monitoring performance of clinical artificial intelligence in health care: a scoping review

Monitoring der klinischen Leistung von künstlicher Intelligenz im Gesundheitswesen: Scoping-Review

Eline Sandvig Andersen iD, Johan Baden Birk-Korch iD, Rasmus Søgaard Hansen iD, Line Haugaard Fly iD, Richard Röttger iD, Diana Maria Cespedes Arcani, Claus Lohman Brasen iD, Ivan Brandslund iD, Jonna Skov Madsen iD

JBI Evidence Synthesis · 2024-12 · 22 (12) · 2423-2446

DOI: 10.11124/jbies-24-00042 ↗

Auf einen Blick

Der Review beschreibt Methoden zum Monitoring klinischer KI-Systeme. Es zeigte sich eine geringe Evidenz- und Leitlinienlage für die praktische Umsetzung.

Fragestellung

Welche Methoden zum Monitoring der klinischen Leistung von KI-Systemen im Gesundheitswesen werden beschrieben, getestet oder implementiert?

Methode

Scoping-Review mit systematischen Datenbanksuchen in MEDLINE, Embase, Scopus und ProQuest Dissertations and Theses Global sowie Zitations- und Grauliteratursuche.

Ergebnisse

39 Evidenzquellen wurden eingeschlossen. Am häufigsten waren Meinungsartikel/narrative Reviews (33 %) und Simulationsstudien (33 %); eine Leitlinie lieferte nur begrenzte Orientierung.

Einschränkungen

Nur eine Leitlinie; wenig konkrete Anleitung zu spezifischen Metriken und statistischen Methoden.

Schlussfolgerung der Autor:innen

Der Review fasst Methoden zum Monitoring von KI im Gesundheitswesen zusammen und zeigt einen Mangel an Evidenz und praktischer Anleitung; weiterer Forschungs- und Diskussionsbedarf besteht.

Kernaussagen

  • 39 Quellen zu Monitoring-Methoden für klinische KI
  • Am häufigsten: AUROC, Sensitivität, Spezifität und prädiktive Werte
  • Wenig spezifische Evidenz und Leitlinien für die Praxis

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Scoping Review in der Zeitschrift JBI Evidence Synthesis, 2024-12. Verfasst von Eline Sandvig Andersen und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biochemistry and Immunology, Lillebaelt Hospital - University Hospital of Southern Denmark, Vejle, Denmark. Hauptthemen laut MEDLINE: Artificial Intelligence. Die Arbeit wurde bisher 71-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

The objective of this review was to provide an overview of the diverse methods described, tested, or implemented for monitoring performance of clinical artificial intelligence (AI) systems, while also summarizing the arguments given for or against these methods.

Introduction

The integration of AI in clinical decision-making is steadily growing. Performances of AI systems evolve over time, necessitating ongoing performance monitoring. However, the evidence on specific monitoring methods is sparse and heterogeneous. Thus, an overview of the evidence on this topic is warranted to guide further research on clinical AI monitoring. Inclusion criteria: We included publications detailing metrics or statistical processes employed in systematic, continuous, or repeated initiatives aimed at evaluating or predicting the clinical performance of AI models with direct implications for patient management in health care. No limitations on language or publication date were enforced.

Methods

We performed systematic database searches in MEDLINE (Ovid), Embase (Ovid), Scopus, and ProQuest Dissertations and Theses Global, supplemented by backward and forward citation searches and gray literature searches. Two or more independent reviewers conducted title and abstract screening, full-text evaluation, and data extraction using a tool developed by the authors. During extraction, the methods identified were divided into subcategories. The results are presented narratively and summarized in tables and graphs.

Results

Thirty-nine sources of evidence were included in the review, with the most abundant source types being opinion papers/narrative reviews (33%) and simulation studies (33%). One guideline on the topic was identified, offering limited guidance on specific metrics and statistical methods. The number of sources included increased year by year, with almost 4 times as many sources included in 2023 compared with 2019. The most commonly reported performance metrics were traditional metrics from the medical literature, including area under the receiver operating characteristics curve (AUROC), sensitivity, specificity, and predictive values, although few arguments were given supporting these choices. Some studies reported on metrics and statistical processing specifically designed to monitor clinical AI.

Conclusion

This review provides a summary of the methods described for monitoring AI in health care. It reveals a relative scarcity of evidence and guidance for specific practical implementation of performance monitoring of clinical AI. This underscores the imperative for further research, discussion, and guidance regarding the specifics of implementing monitoring for clinical AI. The steady increase in the number of relevant sources published per year suggests that this area of research is gaining increased focus, and the amount of evidence and guidance available will likely increase significantly over the coming years. Review registration: Open Science Framework https://osf.io/afkrn

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceHumansDelivery of Health CareClinical Decision-Making

Autor:innen und Institutionen

  • Department of Biochemistry and Immunology, Lillebaelt Hospital - University Hospital of Southern Denmark, Vejle, Denmark.
  • Department of Regional Health Research, University of Southern Denmark, Lillebælt Hospital (Kolding and Vejle), Denmark.
  • Department of Clinical Biochemistry, Odense University Hospital, Odense, Denmark.
  • Department of Mathematics and Computer Science, University of Southern Denmark, Odense, Denmark.
  • Department of Thoracic Surgery, the First Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, China.

ORCID-Profile: Eline Sandvig Andersen, Johan Baden Birk-Korch, Rasmus Søgaard Hansen, Line Haugaard Fly, Richard Röttger, Claus Lohman Brasen, Ivan Brandslund, Jonna Skov Madsen

Zitieren

APA

Andersen, E. S., Birk-Korch, J. B., Hansen, R. S., Fly, L. H., Röttger, R., Arcani, D. M. C., Brasen, C. L., Brandslund, I., & Madsen, J. S. (2024). Monitoring performance of clinical artificial intelligence in health care: a scoping review. JBI Evidence Synthesis, 22(12), 2423-2446. https://doi.org/10.11124/jbies-24-00042

Vancouver

Andersen ES, Birk-Korch JB, Hansen RS, Fly LH, Röttger R, Arcani DMC, et al. Monitoring performance of clinical artificial intelligence in health care: a scoping review. JBI Evidence Synthesis. 2024;22(12):2423-2446. doi:10.11124/jbies-24-00042

BibTeX
@article{Andersen2024,
  author = {Andersen, Eline Sandvig and Birk-Korch, Johan Baden and Hansen, Rasmus Søgaard and Fly, Line Haugaard and Röttger, Richard and Arcani, Diana Maria Cespedes and Brasen, Claus Lohman and Brandslund, Ivan and Madsen, Jonna Skov},
  title = {Monitoring performance of clinical artificial intelligence in health care: a scoping review},
  journal = {JBI Evidence Synthesis},
  year = {2024},
  volume = {22},
  number = {12},
  pages = {2423--2446},
  doi = {10.11124/jbies-24-00042}
}
Kostenlos starten →