Calibrating machine learning approaches for probability estimation: A comprehensive comparison
Statistics in Medicine · 20.12.2023 · 42 (29) · 5451-5478
DOI: 10.1002/sim.9921 ↗Kurz gefasst
Übersichtsarbeit in der Zeitschrift Statistics in Medicine, online erschienen am 17.10.2023. Verfasst von Francisco M Ojeda und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany. Hauptthemen laut MEDLINE: Machine Learning, Models, Statistical. Die Arbeit wurde bisher 81-mal zitiert (Crossref) und nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Statistical prediction models have gained popularity in applied research. One challenge is the transfer of the prediction model to a different population which may be structurally different from the model for which it has been developed. An adaptation to the new population can be achieved by calibrating the model to the characteristics of the target population, for which numerous calibration techniques exist. In view of this diversity, we performed a systematic evaluation of various popular calibration approaches used by the statistical and the machine learning communities for estimating two‐class probabilities. In this work, we first provide a review of the literature and, second, present the results of a comprehensive simulation study. The calibration approaches are compared with respect to their empirical properties and relationships, their ability to generalize precise probability estimates to external populations and their availability in terms of easy‐to‐use software implementations. Third, we provide code from real data analysis allowing its application by researchers. Logistic calibration and beta calibration, which estimate an intercept plus one and two slope parameters, respectively, consistently showed the best results in the simulation studies. Calibration on logit transformed probability estimates generally outperformed calibration methods on nontransformed estimates. In case of structural differences between training and validation data, re‐estimation of the entire prediction model should be outweighted against sample size of the validation data. We recommend regression‐based calibration approaches using transformed probability estimates, where at least one slope is estimated in addition to an intercept for updating probability estimates in validation studies.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
- Centre for Population Health Innovation (POINT), University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
- Cardio-CARE, Medizincampus Davos, Davos, Switzerland.
- German Center for Cardiovascular Research (DZHK), Partner Site Hamburg/Kiel/Lübeck, Hamburg, Germany.
- BDH-Klinik Elzach, Baden-Wuerttemberg, Germany.
- Institute for Medical Informatics, Biometry and Epidemiology, University of Duisburg-Essen, Essen, North Rhine-Westphalia, Germany.
ORCID-Profile: Francisco M Ojeda, Max L Jansen, Matthias Schmid, Andreas Ziegler
Zitieren
APA
Ojeda, F. M., Jansen, M. L., Thiéry, A., Blankenberg, S., Weimar, C., Schmid, M., & Ziegler, A. (2023). Calibrating machine learning approaches for probability estimation: A comprehensive comparison. Statistics in Medicine, 42(29), 5451-5478. https://doi.org/10.1002/sim.9921
Vancouver
Ojeda FM, Jansen ML, Thiéry A, Blankenberg S, Weimar C, Schmid M, et al. Calibrating machine learning approaches for probability estimation: A comprehensive comparison. Statistics in Medicine. 2023;42(29):5451-5478. doi:10.1002/sim.9921
BibTeX
@article{Ojeda2023,
author = {Ojeda, Francisco M and Jansen, Max L and Thiéry, Alexandre and Blankenberg, Stefan and Weimar, Christian and Schmid, Matthias and Ziegler, Andreas},
title = {Calibrating machine learning approaches for probability estimation: A comprehensive comparison},
journal = {Statistics in Medicine},
year = {2023},
volume = {42},
number = {29},
pages = {5451--5478},
doi = {10.1002/sim.9921}
}
EUVIMED