Performance Estimation in Binary Classification Using Calibrated Confidence
Machine Learning · 2026-03 · 115 (3)
DOI: 10.1007/s10994-025-06970-3 ↗Kurz gefasst
Fachartikel in der Zeitschrift Machine Learning, online erschienen am 09.03.2026. Verfasst von Juhani Kivimäki und 3 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: University of Helsinki.
Abstract (Originalsprache)
Model monitoring is a critical component of the machine learning lifecycle, safeguarding against undetected drops in the model’s performance after deployment. Traditionally, performance monitoring has required access to ground truth labels, which are not always readily available. This can result in unacceptable latency or render performance monitoring altogether impossible. Recently, methods designed to estimate the accuracy of classifier models without access to labels have shown promising results. However, there are various other metrics that might be more suitable for assessing model performance in many cases. Until now, none of these important metrics has received similar interest from the scientific community. In this work, we address this gap by presenting Confidence-based Performance estimation (CBPE), a novel method that can estimate any binary classification metric defined using the confusion matrix. In particular, we choose four metrics from this large family: accuracy, precision, recall, and $$\hbox {F}_1$$ , to demonstrate our method. CBPE treats the elements of the confusion matrix as random variables and leverages calibrated confidence scores of the model to estimate their distributions. The desired metric is then also treated as a random variable, whose full probability distribution can be derived from the estimated confusion matrix. CBPE is shown to produce estimates that come with strong theoretical guarantees and valid confidence intervals.
Förderung
- University of Helsinki
Zitieren
APA
Kivimäki, J., Białek, J., Kuberski, W., & Nurminen, J. K. (2026). Performance Estimation in Binary Classification Using Calibrated Confidence. Machine Learning, 115(3). https://doi.org/10.1007/s10994-025-06970-3
Vancouver
Kivimäki J, Białek J, Kuberski W, Nurminen JK. Performance Estimation in Binary Classification Using Calibrated Confidence. Machine Learning. 2026;115(3). doi:10.1007/s10994-025-06970-3
BibTeX
@article{Kivimki2026,
author = {Kivimäki, Juhani and Białek, Jakub and Kuberski, Wojtek and Nurminen, Jukka K.},
title = {Performance Estimation in Binary Classification Using Calibrated Confidence},
journal = {Machine Learning},
year = {2026},
volume = {115},
number = {3},
doi = {10.1007/s10994-025-06970-3}
}
EUVIMED