Distribution-free, Risk-controlling Prediction Sets
Journal of the ACM · 31.12.2021 · 68 (6) · 1-34
DOI: 10.1145/3478535 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the ACM, online erschienen am 30.09.2021. Verfasst von Stephen Bates und 4 weiteren Autor:innen. Die Arbeit wurde bisher 145-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation Graduate Research Fellowship Program and a Berkeley Fellowship, Army Research Office.
Abstract (Originalsprache)
While improving prediction accuracy has been the focus of machine learning in recent years, this alone does not suffice for reliable decision-making. Deploying learning systems in consequential settings also requires calibrating and communicating the uncertainty of predictions. To convey instance-wise uncertainty for prediction tasks, we show how to generate set-valued predictions from a black-box predictor that controls the expected loss on future test points at a user-specified level. Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets. This framework enables simple, distribution-free, rigorous error control for many tasks, and we demonstrate it in five large-scale machine learning problems: (1) classification problems where some mistakes are more costly than others; (2) multi-label classification, where each observation has multiple associated labels; (3) classification problems where the labels have a hierarchical structure; (4) image segmentation, where we wish to predict a set of pixels containing an object of interest; and (5) protein structure prediction. Last, we discuss extensions to uncertainty quantification for ranking, metric learning, and distributionally robust learning.
Förderung
- National Science Foundation Graduate Research Fellowship Program and a Berkeley Fellowship
- Army Research Office
Zitieren
APA
Bates, S., Angelopoulos, A., Lei, L., Malik, J., & Jordan, M. (2021). Distribution-free, Risk-controlling Prediction Sets. Journal of the ACM, 68(6), 1-34. https://doi.org/10.1145/3478535
Vancouver
Bates S, Angelopoulos A, Lei L, Malik J, Jordan M. Distribution-free, Risk-controlling Prediction Sets. Journal of the ACM. 2021;68(6):1-34. doi:10.1145/3478535
BibTeX
@article{Bates2021,
author = {Bates, Stephen and Angelopoulos, Anastasios and Lei, Lihua and Malik, Jitendra and Jordan, Michael},
title = {Distribution-free, Risk-controlling Prediction Sets},
journal = {Journal of the ACM},
year = {2021},
volume = {68},
number = {6},
pages = {1--34},
doi = {10.1145/3478535}
}
EUVIMED