EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift

Dustin Wright iD, Isabelle Augenstein

PLOS One · 2025

DOI: 10.1371/journal.pone.0323064 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Dustin Wright und 1 weiteren Autor:innen. Beteiligte Einrichtung: University of Copenhagen, Department of Computer Science, Copenhagen, Denmark. Hauptthemen laut MEDLINE: Machine Learning, Crowdsourcing. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Horizon 2020, Danish Data Science Academy.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Selecting an effective training signal for machine learning tasks is difficult: expert annotations are expensive, and crowd-sourced annotations may not be reliable. Recent work has demonstrated that learning from a distribution over labels acquired from crowd annotations can be effective both for performance and uncertainty estimation. However, this has mainly been studied using a limited set of soft-labeling methods in an in-domain setting. Additionally, no one method has been shown to consistently perform well across tasks, making it difficult to know a priori which to choose. To fill these gaps, this paper provides the first large-scale empirical study on learning from crowd labels in the out-of-domain setting, systematically analyzing 8 soft-labeling methods on 4 language and vision tasks. Additionally, we propose to aggregate soft-labels via a simple average in order to achieve consistent performance across tasks. We demonstrate that this yields classifiers with improved predictive uncertainty estimation in most settings while maintaining consistent raw performance compared to learning from individual soft-labeling methods or taking a majority vote of the annotations. We additionally highlight that in regimes with abundant or minimal training data, the selection of soft labeling method is less important, while for highly subjective labels and moderate amounts of training data, aggregation yields significant improvements in uncertainty estimation over individual methods. Code can be found at https://github.com/copenlu/aggregating-crowd-annotations-ood

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Machine LearningCrowdsourcingUncertaintyHumansAlgorithms

Autor:innen und Institutionen

  • University of Copenhagen, Department of Computer Science, Copenhagen, Denmark.

ORCID-Profile: Dustin Wright

Förderung

  • Horizon 2020
  • Danish Data Science Academy

Zitieren

APA

Wright, D., & Augenstein, I. (2025). Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift. PLOS One. https://doi.org/10.1371/journal.pone.0323064

Vancouver

Wright D, Augenstein I. Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift. PLOS One. 2025. doi:10.1371/journal.pone.0323064

BibTeX
@article{Wright2025,
  author = {Wright, Dustin and Augenstein, Isabelle},
  title = {Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift},
  journal = {PLOS One},
  year = {2025},
  doi = {10.1371/journal.pone.0323064}
}
Kostenlos starten →