EUVIMEDEUVIMED entdecken
Crossref

Confidence-based Estimators for Predictive Performance in Model Monitoring

Juhani Kivimäki, Jukka K. Nurminen, Jakub Białek, Wojtek Kuberski

Journal of Artificial Intelligence Research · 20.01.2025 · 82 · 209-240

DOI: 10.1613/jair.1.16709 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Artificial Intelligence Research, online erschienen am 20.01.2025. Verfasst von Juhani Kivimäki und 3 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

After a machine learning model has been deployed into production, its predictive performance needs to be monitored. Ideally, such monitoring can be carried out by comparing the model’s predictions against ground truth labels. For this to be possible, the ground truth labels must be available relatively soon after inference. However, there are many use cases where ground truth labels are available only after a significant delay, or in the worst case, not at all. In such cases, directly monitoring the model’s predictive performance is impossible. Recently, novel methods for estimating the predictive performance of a model when ground truth is unavailable have been developed. Many of these methods leverage model confidence or other uncertainty estimates and are experimentally compared against a naive baseline method, namely Average Confidence (AC), which estimates model accuracy as the average of confidence scores for a given set of predictions. However, until now the theoretical properties of the AC method have not been properly explored. In this paper, we bridge this gap by reviewing the AC method and show that under certain general assumptions, it is an unbiased and consistent estimator of model accuracy. We also augment the AC method by deriving valid confidence intervals for the estimates it produces. These contributions elevate AC from an ad-hoc estimator to a principled one, encouraging its use in practice. We complement our theoretical results with empirical experiments, comparing AC against more complex estimators in a monitoring setting under covariate shift. We conduct our experiments using synthetic datasets, which allow for full control over the nature of the shift. Our experiments with binary classifiers show that the AC method is able to beat other estimators in many cases. However, the comparative quality of the different estimators is found to be heavily case-dependent.

Zitieren

APA

Kivimäki, J., Nurminen, J. K., Białek, J., & Kuberski, W. (2025). Confidence-based Estimators for Predictive Performance in Model Monitoring. Journal of Artificial Intelligence Research, 82, 209-240. https://doi.org/10.1613/jair.1.16709

Vancouver

Kivimäki J, Nurminen JK, Białek J, Kuberski W. Confidence-based Estimators for Predictive Performance in Model Monitoring. Journal of Artificial Intelligence Research. 2025;82:209-240. doi:10.1613/jair.1.16709

BibTeX
@article{Kivimki2025,
  author = {Kivimäki, Juhani and Nurminen, Jukka K. and Białek, Jakub and Kuberski, Wojtek},
  title = {Confidence-based Estimators for Predictive Performance in Model Monitoring},
  journal = {Journal of Artificial Intelligence Research},
  year = {2025},
  volume = {82},
  pages = {209--240},
  doi = {10.1613/jair.1.16709}
}
Kostenlos starten →