EUVIMEDOntdek EUVIMED
Open access · CC BY-NC 4.0

Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data

Eleonora Di Carluccio iD, Georgios Koliopanos iD, Francisco M Ojeda, Christian Weimar, Andreas Ziegler iD

Statistics in Medicine · 2026

DOI: 10.1002/sim.70751 ↗

At a glance

Journal article in Statistics in Medicine, 2026. Written by Eleonora Di Carluccio and 4 co-authors. Institution: Cardio-CARE, Medizincampus Davos, Davos, Switzerland. Major MEDLINE topics: Machine Learning, Models, Statistical. 46 references. The full text is freely available (open access, CC BY-NC 4.0).

SamenvattingSubjectsBibliografische gegevensCiteRelated studies

Abstract (original language)

Statistical prediction models for binary outcomes are becoming increasingly popular. One significant challenge is calibrating these models to suit the characteristics of a target population that is structurally different from the original population. Calibration is especially challenging when there is no training data available from the target population. To address this problem, we propose a novel calibration method, SimCal, which uses synthetic data generated from the model development data in conjunction with marginal statistics from the calibration cohort. We show that expert judgment modeling (EJM) may be used for calibration if cross‐sectional data from the target population are available comprising expert judgments about the potential outcome and the covariates. We describe three alternative calibration approaches when calibration data are lacking: similarity‐binning averaging (SBA), adaptive calibration of predictions (ACP), and Elkan calibration. In a simulation study, we compare SBA, ACP, Elkan calibration, and SimCal. R code for applying these methods is provided from the re‐analysis of data on coronary artery disease. We illustrate all 5 calibration approaches with a real data set for predicting functional outcome after stroke and all approaches but EJM in the re‐analysis of the Cleveland Clinic data. None of the approaches performed convincingly well in all situations. SimCal performed well when model parameters were correctly specified. EJM failed on the stroke data. Further research is urgently required for calibration in the absence of calibration data.

Subjects

MeSH terms (MEDLINE)

Machine LearningModels, StatisticalCalibrationHumansComputer SimulationProbabilityStrokePredictive Learning ModelsCoronary Artery DiseasePrediction AlgorithmsCross-Sectional Studies

Author keywords

Elkan calibrationadaptive calibration of predictionsexpert judgment modelingjudgmental bootstrapsimilarity binning averaging

Authors and institutions

  • Cardio-CARE, Medizincampus Davos, Davos, Switzerland.
  • Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
  • Center for Population Health Innovation (POINT), University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
  • Partner Site Hamburg/Kiel/Lübeck, German Center for Cardiovascular Research (DZHK), Hamburg, Germany.
  • BDH-Klinik Elzach, Elzach, Germany.
  • Institute for Medical Informatics, Biometry and Epidemiology, University Hospital, University Duisburg-Essen, Essen, Germany.

ORCID profiles: Eleonora Di Carluccio, Georgios Koliopanos, Andreas Ziegler

Cite

APA

Carluccio, E. D., Koliopanos, G., Ojeda, F. M., Weimar, C., & Ziegler, A. (2026). Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data. Statistics in Medicine. https://doi.org/10.1002/sim.70751

Vancouver

Carluccio ED, Koliopanos G, Ojeda FM, Weimar C, Ziegler A. Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data. Statistics in Medicine. 2026. doi:10.1002/sim.70751

BibTeX
@article{Carluccio2026,
  author = {Carluccio, Eleonora Di and Koliopanos, Georgios and Ojeda, Francisco M and Weimar, Christian and Ziegler, Andreas},
  title = {Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data},
  journal = {Statistics in Medicine},
  year = {2026},
  doi = {10.1002/sim.70751}
}
Start for free →