Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift
PLOS One · 2025
DOI: 10.1371/journal.pone.0323064 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Dustin Wright und 1 weiteren Autor:innen. Beteiligte Einrichtung: University of Copenhagen, Department of Computer Science, Copenhagen, Denmark. Hauptthemen laut MEDLINE: Machine Learning, Crowdsourcing. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Horizon 2020, Danish Data Science Academy.
Abstract (Originalsprache)
Selecting an effective training signal for machine learning tasks is difficult: expert annotations are expensive, and crowd-sourced annotations may not be reliable. Recent work has demonstrated that learning from a distribution over labels acquired from crowd annotations can be effective both for performance and uncertainty estimation. However, this has mainly been studied using a limited set of soft-labeling methods in an in-domain setting. Additionally, no one method has been shown to consistently perform well across tasks, making it difficult to know a priori which to choose. To fill these gaps, this paper provides the first large-scale empirical study on learning from crowd labels in the out-of-domain setting, systematically analyzing 8 soft-labeling methods on 4 language and vision tasks. Additionally, we propose to aggregate soft-labels via a simple average in order to achieve consistent performance across tasks. We demonstrate that this yields classifiers with improved predictive uncertainty estimation in most settings while maintaining consistent raw performance compared to learning from individual soft-labeling methods or taking a majority vote of the annotations. We additionally highlight that in regimes with abundant or minimal training data, the selection of soft labeling method is less important, while for highly subjective labels and moderate amounts of training data, aggregation yields significant improvements in uncertainty estimation over individual methods. Code can be found at https://github.com/copenlu/aggregating-crowd-annotations-ood
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- University of Copenhagen, Department of Computer Science, Copenhagen, Denmark.
ORCID-Profile: Dustin Wright
Förderung
- Horizon 2020
- Danish Data Science Academy
Zitieren
APA
Wright, D., & Augenstein, I. (2025). Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift. PLOS One. https://doi.org/10.1371/journal.pone.0323064
Vancouver
Wright D, Augenstein I. Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift. PLOS One. 2025. doi:10.1371/journal.pone.0323064
BibTeX
@article{Wright2025,
author = {Wright, Dustin and Augenstein, Isabelle},
title = {Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift},
journal = {PLOS One},
year = {2025},
doi = {10.1371/journal.pone.0323064}
}
EUVIMED