EUVIMEDUpptäck EUVIMED
Open access · CC BY-NC-ND 4.0

Instability of the AUROC of Clinical Prediction Models

Florian D van Leeuwen iD, Ewout W Steyerberg iD, David van Klaveren iD, Ben Wessler, David M Kent, Erik W van Zwet iD

Statistics in Medicine · 2025-02-28 · 44 (5)

DOI: 10.1002/sim.70011 ↗

At a glance

Journal article in Statistics in Medicine, published online on 2025-02-08. Written by Florian D van Leeuwen and 5 co-authors. Institution: Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands. Major MEDLINE topics: Area Under Curve, Models, Statistical, ROC Curve. Cited 20 times so far (Crossref); 31 references. The full text is freely available (open access, CC BY-NC-ND 4.0). Funded by: NIA NIH HHS.

AbstractSubjectsBibliografiska uppgifterCiteRelated studies

Abstract (original language)

Background

External validations are essential to assess the performance of a clinical prediction model (CPM) before deployment. Apart from model misspecification, also differences in patient population, the standard of care, predictor definitions, and other factors influence a model's discriminative ability, as commonly quantified by the AUC (or c‐statistic). We aimed to quantify the variation in AUCs across sets of external validation studies and propose ways to adjust expectations of a model's performance in a new setting.

Methods

The Tufts‐PACE CPM Registry holds a collection of CPMs for prognosis in cardiovascular disease. We analyzed the AUC estimates of 469 CPMs with at least one external validation. Combined, these CPMs had a total of 1603 external validations reported in the literature. For each CPM and its associated set of validation studies, we performed a random‐effects meta‐analysis to estimate the between‐study standard deviation among the AUCs. Since the majority of these meta‐analyses have only a handful of validations, this leads to very poor estimates of . So, instead of focusing on a single CPM, we estimated a log‐normal distribution of across all 469 CPMs. We then used this distribution as an empirical prior. We used cross‐validation to compare this empirical Bayesian approach with frequentist fixed and random‐effects meta‐analyses.

Results

The 469 CPMs included in our study had a median of 2 external validations with an IQR of [1–3]. The estimated distribution of had a mean of 0.055 and a standard deviation of 0.015. If = 0.05, then the 95% prediction interval for the AUC in a new setting has a width of at least 0.1, no matter how many validations have been done. When there are fewer than 5 validations, which is typically the case, the usual frequentist methods grossly underestimate the uncertainty about the AUC in a new setting. Accounting for in a Bayesian approach achieved near nominal coverage.

Conclusion

Due to large heterogeneity among the validated AUC values of a CPM, there is great irreducible uncertainty in predicting the AUC in a new setting. This uncertainty is underestimated by existing methods. The proposed empirical Bayes approach addresses this problem which merits wide application in judging the validity of prediction models.

Subjects

MeSH terms (MEDLINE)

Area Under CurveModels, StatisticalROC CurveHumansCardiovascular DiseasesRegistriesPrognosisBayes Theorem

Author keywords

CPMclinical prediction modelsempirical Bayesheterogeneitymeta‐analysis

Authors and institutions

  • Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.
  • Department of Public Health, Erasmus University Medical Center, Rotterdam, Netherlands.
  • Predictive Analytics and Comparative Effectiveness Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, Massachusetts, USA.

ORCID profiles: Florian D van Leeuwen, Ewout W Steyerberg, David van Klaveren, Erik W van Zwet

Funding

  • NIA NIH HHS

Cite

APA

van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025). Instability of the AUROC of Clinical Prediction Models. Statistics in Medicine, 44(5). https://doi.org/10.1002/sim.70011

Vancouver

van Leeuwen FD, Steyerberg EW, van Klaveren D, Wessler B, Kent DM, van Zwet EW. Instability of the AUROC of Clinical Prediction Models. Statistics in Medicine. 2025;44(5). doi:10.1002/sim.70011

BibTeX
@article{vanLeeuwen2025,
  author = {van Leeuwen, Florian D and Steyerberg, Ewout W and van Klaveren, David and Wessler, Ben and Kent, David M and van Zwet, Erik W},
  title = {Instability of the AUROC of Clinical Prediction Models},
  journal = {Statistics in Medicine},
  year = {2025},
  volume = {44},
  number = {5},
  doi = {10.1002/sim.70011}
}
Start for free →