Critical Appraisal Tools for Evaluating Artificial Intelligence in Clinical Studies: Scoping Review
Werkzeuge zur kritischen Bewertung von künstlicher Intelligenz in klinischen Studien: Scoping Review
Journal of Medical Internet Research · 08.12.2025 · 27 · e77110-e77110
DOI: 10.2196/77110 ↗Auf einen Blick
Die Übersichtsarbeit identifizierte 46 Werkzeuge zur Bewertung von KI-Studien; am häufigsten fanden sich Berichtsleitfäden. Werkzeuge zur kritischen Bewertung und zum Bias waren seltener.
Fragestellung
Welche Werkzeuge zur kritischen Bewertung klinischer Studien mit KI gibt es, und welche Konzepte und Domänen erfassen sie?
Methode
Scoping Review; Suche in MEDLINE, Embase, CINAHL, PsycINFO und IEEE von Beginn bis April 2024; PRISMA-ScR und PRISMA-S.
Ergebnisse
70 Datensätze eingeschlossen. 46 Werkzeuge: 26 Leitfäden für das Reporting von KI-Studien, 16 Tools zur kritischen Bewertung, 2 zur Studienqualität, 2 zum Risk of Bias. Außerdem 9 Arbeiten zur Bias-Klassifikation/-Minderung und 15 Chatbot-Assessment-Studien oder -Reviews.
Schlussfolgerung der Autor:innen
Die Autoren beschreiben eine Landschaft von Evidenzwerkzeugen mit Schwerpunkt auf Reporting-Tools, gefolgt von kritischer Bewertung und wenigen Bias-Tools. Für die Bewertung sollten Fairness und Bias-Minderung in KI stärker berücksichtigt werden.
Kernaussagen
- 46 Werkzeuge zur Bewertung von KI-Studien identifiziert
- Reporting-Leitfäden überwogen gegenüber Appraisal-Tools
- Chatbot-Assessment ist ein heterogenes und wachsendes Feld
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Scoping Review in der Zeitschrift Journal of Medical Internet Research, online erschienen am 08.12.2025. Verfasst von Juan B Cabello und 11 weiteren Autor:innen. Beteiligte Einrichtung: Critical Appraisal Skills Program Spain, C/ Enriqueta Elizaizin, 2, E 5, 7C, Alicante, 03007, Spain, 34 619669243. Hauptthemen laut MEDLINE: Artificial Intelligence. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 102 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Health research that uses predictive and generative artificial intelligence (AI) is rapidly growing. As in traditional clinical studies, the way in which AI studies are conducted can introduce systematic errors. The translation of this AI evidence into clinical practice and research needs critical appraisal tools for clinical decision-makers and researchers.
Objective
This study aimed to identify existing tools for the critical appraisal of clinical studies that use AI and to examine the concepts and domains these tools explore. The research question was framed using the Population-Concept-Context (PCC) framework. Population (P): AI clinical studies; Concept (C): tools for critical appraisal and associated constructs such as quality, reporting, validity, risk of bias, and applicability; and context (C): clinical practice. In addition, studies on bias classification and chatbot assessment were included.
Methods
We searched medical and engineering databases (MEDLINE, Embase, CINAHL, PsycINFO, and IEEE) from inception to April 2024. We included clinical primary research with tools for critical appraisal. Classical reviews and systematic reviews were included in the first phase of screening and excluded in the secondary phase after identifying new tools by forward snowballing. We excluded nonhuman, computer, and mathematical research, and letters, opinion papers, and editorials. We used Rayyan (Qatar Computing Research Institute) for screening. Data extraction was done by two reviewers, and discrepancies were resolved through discussion. The protocol was previously registered in Open Science Framework . We adhered to the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) and the PRISMA-S (PRISMA-Search) extension for reporting literature in systematic reviews.
Results
We retrieved 4393 unique records for screening. After excluding 3803 records, 119 were selected for full-text screening. From these, 59 were excluded. After inclusion of 10 studies via other methods, a total of 70 records were finally included. We found 46 tools (26 guides for reporting AI studies, 16 tools for critical appraisal, 2 for study quality, and 2 for risk of bias). Nine papers focused on bias classification or mitigation. We found 15 chatbot assessment studies or systematic reviews of chatbot studies (6 and 9, respectively), which are a very heterogeneous group.
Conclusions
The results picture a landscape of evidence tools where reporting tools predominate, followed by critical appraisal, and a few tools for risk of bias. The mismatch of bias in AI and epidemiology should be considered for critical appraisal, especially regarding fairness and bias mitigation in AI. Finally, chatbot assessment studies represent a vast and evolving field in which progress in design, reporting, and critical appraisal is necessary and urgent.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Critical Appraisal Skills Program Spain, C/ Enriqueta Elizaizin, 2, E 5, 7C, Alicante, 03007, Spain, 34 619669243.
- Unidad de Hospitalización a Domicilio, Hospital Universitari i Politècnic La Fe, Valencia, Spain.
- Servicio de Medicina Interna, Hospital Universitario de Guadalajara, Guadalajara, Spain.
- Department of ENT, Hospital Vital Alvarez Buylla, Mieres, Spain.
- Hospital Donostia, Donostia - San Sebastian, Spain.
- Biblioteca virtual de salud de Euskadi, Vitoria, Spain.
ORCID-Profile: Juan B Cabello, Vicente Ruiz Garcia, Miguel Torralba, Miguel Maldonado Fernandez, Marimar Ubeda, Eukene Ansuategui, Luis Ramos-Ruperto, Jose I Emparanza, Iratxe Urreta, Maria Teresa Iglesias, Jose I Pijoan, Amanda Burls
Zitieren
APA
Cabello, J. B., Ruiz Garcia, V., Torralba, M., Maldonado Fernandez, M., Ubeda, M., Ansuategui, E., Ramos-Ruperto, L., Emparanza, J. I., Urreta, I., Iglesias, M. T., Pijoan, J. I., & Burls, A. (2025). Critical Appraisal Tools for Evaluating Artificial Intelligence in Clinical Studies: Scoping Review. Journal of Medical Internet Research, 27, e77110-e77110. https://doi.org/10.2196/77110
Vancouver
Cabello JB, Ruiz Garcia V, Torralba M, Maldonado Fernandez M, Ubeda M, Ansuategui E, et al. Critical Appraisal Tools for Evaluating Artificial Intelligence in Clinical Studies: Scoping Review. Journal of Medical Internet Research. 2025;27:e77110-e77110. doi:10.2196/77110
BibTeX
@article{Cabello2025,
author = {Cabello, Juan B and Ruiz Garcia, Vicente and Torralba, Miguel and Maldonado Fernandez, Miguel and Ubeda, Marimar and Ansuategui, Eukene and Ramos-Ruperto, Luis and Emparanza, Jose I and Urreta, Iratxe and Iglesias, Maria Teresa and Pijoan, Jose I and Burls, Amanda},
title = {Critical Appraisal Tools for Evaluating Artificial Intelligence in Clinical Studies: Scoping Review},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e77110--e77110},
doi = {10.2196/77110}
}
EUVIMED