EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Distribution-free, Risk-controlling Prediction Sets

Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, Michael Jordan

Journal of the ACM · 31.12.2021 · 68 (6) · 1-34

DOI: 10.1145/3478535 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the ACM, online erschienen am 30.09.2021. Verfasst von Stephen Bates und 4 weiteren Autor:innen. Die Arbeit wurde bisher 145-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation Graduate Research Fellowship Program and a Berkeley Fellowship, Army Research Office.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While improving prediction accuracy has been the focus of machine learning in recent years, this alone does not suffice for reliable decision-making. Deploying learning systems in consequential settings also requires calibrating and communicating the uncertainty of predictions. To convey instance-wise uncertainty for prediction tasks, we show how to generate set-valued predictions from a black-box predictor that controls the expected loss on future test points at a user-specified level. Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets. This framework enables simple, distribution-free, rigorous error control for many tasks, and we demonstrate it in five large-scale machine learning problems: (1) classification problems where some mistakes are more costly than others; (2) multi-label classification, where each observation has multiple associated labels; (3) classification problems where the labels have a hierarchical structure; (4) image segmentation, where we wish to predict a set of pixels containing an object of interest; and (5) protein structure prediction. Last, we discuss extensions to uncertainty quantification for ranking, metric learning, and distributionally robust learning.

Förderung

  • National Science Foundation Graduate Research Fellowship Program and a Berkeley Fellowship
  • Army Research Office

Zitieren

APA

Bates, S., Angelopoulos, A., Lei, L., Malik, J., & Jordan, M. (2021). Distribution-free, Risk-controlling Prediction Sets. Journal of the ACM, 68(6), 1-34. https://doi.org/10.1145/3478535

Vancouver

Bates S, Angelopoulos A, Lei L, Malik J, Jordan M. Distribution-free, Risk-controlling Prediction Sets. Journal of the ACM. 2021;68(6):1-34. doi:10.1145/3478535

BibTeX
@article{Bates2021,
  author = {Bates, Stephen and Angelopoulos, Anastasios and Lei, Lihua and Malik, Jitendra and Jordan, Michael},
  title = {Distribution-free, Risk-controlling Prediction Sets},
  journal = {Journal of the ACM},
  year = {2021},
  volume = {68},
  number = {6},
  pages = {1--34},
  doi = {10.1145/3478535}
}
Kostenlos starten →