EUVIMEDEUVIMED entdecken
Systematische ÜbersichtOpen Access

APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support

Jethro C C Kwong, Adree Khondker, Katherine Lajkosz, Matthew B A McDermott, Xavier Borrat Frigola, Melissa D McCradden, Muhammad Mamdani, Girish S Kulkarni, Alistair E W Johnson

JAMA Network Open · 25.09.2023 · 6 (9) · e2335377

DOI: 10.1001/jamanetworkopen.2023.35377 ↗

Kurz gefasst

Systematische Übersicht in der Zeitschrift JAMA Network Open, online erschienen am 25.09.2023. Verfasst von Jethro C C Kwong und 8 weiteren Autor:innen. Beteiligte Einrichtung: Division of Urology, Department of Surgery, University of Toronto, Toronto, Ontario, Canada. Hauptthemen laut MEDLINE: Artificial Intelligence, Decision Support Systems, Clinical. Die Arbeit wurde bisher 152-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Importance Artificial intelligence (AI) has gained considerable attention in health care, yet concerns have been raised around appropriate methods and fairness. Current AI reporting guidelines do not provide a means of quantifying overall quality of AI research, limiting their ability to compare models addressing the same clinical question.

Objective

To develop a tool (APPRAISE-AI) to evaluate the methodological and reporting quality of AI prediction models for clinical decision support. Design, Setting, and Participants This quality improvement study evaluated AI studies in the model development, silent, and clinical trial phases using the APPRAISE-AI tool, a quantitative method for evaluating quality of AI studies across 6 domains: clinical relevance, data quality, methodological conduct, robustness of results, reporting quality, and reproducibility. These domains included 24 items with a maximum overall score of 100 points. Points were assigned to each item, with higher points indicating stronger methodological or reporting quality. The tool was applied to a systematic review on machine learning to estimate sepsis that included articles published until September 13, 2019. Data analysis was performed from September to December 2022. Main Outcomes and Measures The primary outcomes were interrater and intrarater reliability and the correlation between APPRAISE-AI scores and expert scores, 3-year citation rate, number of Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) low risk-of-bias domains, and overall adherence to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement.

Results

A total of 28 studies were included. Overall APPRAISE-AI scores ranged from 33 (low quality) to 67 (high quality). Most studies were moderate quality. The 5 lowest scoring items included source of data, sample size calculation, bias assessment, error analysis, and transparency. Overall APPRAISE-AI scores were associated with expert scores (Spearman ρ, 0.82; 95% CI, 0.64-0.91; P < .001), 3-year citation rate (Spearman ρ, 0.69; 95% CI, 0.43-0.85; P < .001), number of QUADAS-2 low risk-of-bias domains (Spearman ρ, 0.56; 95% CI, 0.24-0.77; P = .002), and adherence to the TRIPOD statement (Spearman ρ, 0.87; 95% CI, 0.73-0.94; P < .001). Intraclass correlation coefficient ranges for interrater and intrarater reliability were 0.74 to 1.00 for individual items, 0.81 to 0.99 for individual domains, and 0.91 to 0.98 for overall scores. Conclusions and Relevance In this quality improvement study, APPRAISE-AI demonstrated strong interrater and intrarater reliability and correlated well with several study quality measures. This tool may provide a quantitative approach for investigators, reviewers, editors, and funding organizations to compare the research quality across AI studies for clinical decision support.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceDecision Support Systems, ClinicalHumansReproducibility of ResultsMachine LearningClinical Relevance

Autor:innen und Institutionen

  • Division of Urology, Department of Surgery, University of Toronto, Toronto, Ontario, Canada.
  • Temerty Centre for AI Research and Education in Medicine, University of Toronto, Toronto, Ontario, Canada.
  • Department of Biostatistics, University Health Network, University of Toronto, Toronto, Ontario, Canada.
  • Department of Biomedical Informatics, Massachusetts Institute of Technology, Cambridge.
  • Laboratory for Computational Physiology, Harvard-Massachusetts Institute of Technology Division of Health Sciences and Technology, Cambridge.
  • Anesthesiology and Critical Care Department, Hospital Clinic de Barcelona, Barcelona, Spain.

Zitieren

APA

Kwong, J. C. C., Khondker, A., Lajkosz, K., McDermott, M. B. A., Frigola, X. B., McCradden, M. D., Mamdani, M., Kulkarni, G. S., & Johnson, A. E. W. (2023). APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support. JAMA Network Open, 6(9), e2335377. https://doi.org/10.1001/jamanetworkopen.2023.35377

Vancouver

Kwong JCC, Khondker A, Lajkosz K, McDermott MBA, Frigola XB, McCradden MD, et al. APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support. JAMA Network Open. 2023;6(9):e2335377. doi:10.1001/jamanetworkopen.2023.35377

BibTeX
@article{Kwong2023,
  author = {Kwong, Jethro C C and Khondker, Adree and Lajkosz, Katherine and McDermott, Matthew B A and Frigola, Xavier Borrat and McCradden, Melissa D and Mamdani, Muhammad and Kulkarni, Girish S and Johnson, Alistair E W},
  title = {APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support},
  journal = {JAMA Network Open},
  year = {2023},
  volume = {6},
  number = {9},
  pages = {e2335377},
  doi = {10.1001/jamanetworkopen.2023.35377}
}
Kostenlos starten →