EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Instability of the AUROC of Clinical Prediction Models

Florian D van Leeuwen iD, Ewout W Steyerberg iD, David van Klaveren iD, Ben Wessler, David M Kent, Erik W van Zwet iD

Statistics in Medicine · 28.02.2025 · 44 (5)

DOI: 10.1002/sim.70011 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Statistics in Medicine, online erschienen am 08.02.2025. Verfasst von Florian D van Leeuwen und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands. Hauptthemen laut MEDLINE: Area Under Curve, Models, Statistical, ROC Curve. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NIA NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

External validations are essential to assess the performance of a clinical prediction model (CPM) before deployment. Apart from model misspecification, also differences in patient population, the standard of care, predictor definitions, and other factors influence a model's discriminative ability, as commonly quantified by the AUC (or c‐statistic). We aimed to quantify the variation in AUCs across sets of external validation studies and propose ways to adjust expectations of a model's performance in a new setting.

Methods

The Tufts‐PACE CPM Registry holds a collection of CPMs for prognosis in cardiovascular disease. We analyzed the AUC estimates of 469 CPMs with at least one external validation. Combined, these CPMs had a total of 1603 external validations reported in the literature. For each CPM and its associated set of validation studies, we performed a random‐effects meta‐analysis to estimate the between‐study standard deviation among the AUCs. Since the majority of these meta‐analyses have only a handful of validations, this leads to very poor estimates of . So, instead of focusing on a single CPM, we estimated a log‐normal distribution of across all 469 CPMs. We then used this distribution as an empirical prior. We used cross‐validation to compare this empirical Bayesian approach with frequentist fixed and random‐effects meta‐analyses.

Results

The 469 CPMs included in our study had a median of 2 external validations with an IQR of [1–3]. The estimated distribution of had a mean of 0.055 and a standard deviation of 0.015. If = 0.05, then the 95% prediction interval for the AUC in a new setting has a width of at least 0.1, no matter how many validations have been done. When there are fewer than 5 validations, which is typically the case, the usual frequentist methods grossly underestimate the uncertainty about the AUC in a new setting. Accounting for in a Bayesian approach achieved near nominal coverage.

Conclusion

Due to large heterogeneity among the validated AUC values of a CPM, there is great irreducible uncertainty in predicting the AUC in a new setting. This uncertainty is underestimated by existing methods. The proposed empirical Bayes approach addresses this problem which merits wide application in judging the validity of prediction models.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Area Under CurveModels, StatisticalROC CurveHumansCardiovascular DiseasesRegistriesPrognosisBayes Theorem

Stichwörter der Autor:innen

CPMclinical prediction modelsempirical Bayesheterogeneitymeta‐analysis

Autor:innen und Institutionen

  • Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.
  • Department of Public Health, Erasmus University Medical Center, Rotterdam, Netherlands.
  • Predictive Analytics and Comparative Effectiveness Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, Massachusetts, USA.

ORCID-Profile: Florian D van Leeuwen, Ewout W Steyerberg, David van Klaveren, Erik W van Zwet

Förderung

  • NIA NIH HHS

Zitieren

APA

van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025). Instability of the AUROC of Clinical Prediction Models. Statistics in Medicine, 44(5). https://doi.org/10.1002/sim.70011

Vancouver

van Leeuwen FD, Steyerberg EW, van Klaveren D, Wessler B, Kent DM, van Zwet EW. Instability of the AUROC of Clinical Prediction Models. Statistics in Medicine. 2025;44(5). doi:10.1002/sim.70011

BibTeX
@article{vanLeeuwen2025,
  author = {van Leeuwen, Florian D and Steyerberg, Ewout W and van Klaveren, David and Wessler, Ben and Kent, David M and van Zwet, Erik W},
  title = {Instability of the AUROC of Clinical Prediction Models},
  journal = {Statistics in Medicine},
  year = {2025},
  volume = {44},
  number = {5},
  doi = {10.1002/sim.70011}
}
Kostenlos starten →