EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks

Tadesse Destaw Belay, Ibrahim Said Ahmad, Idris Abdulmumin, Abinew Ali Ayele, Alexander Gelbukh, Eusebio Ricárdez-Vázquez, Olga Kolesnikova, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam

Transactions of the Association for Computational Linguistics · 01.07.2026 · 14 · 1329-1347

DOI: 10.1162/tacl.a.731 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.07.2026. Verfasst von Tadesse Destaw Belay und 8 weiteren Autor:innen. Die Arbeit nennt 58 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has explored modeling individual annotators to preserve their perspectives. However, modeling each annotator is resource-intensive and remains underexplored across various NLP tasks. We propose an agreement-based clustering technique to model the disagreement between the annotators. We conduct comprehensive experiments in 40 datasets in 18 typologically diverse languages, covering three subjective NLP tasks: sentiment analysis, emotion classification, and hate speech detection. We evaluate four aggregation approaches: majority vote, ensemble, multi-label, and multitask. The results demonstrate that agreement-based clustering can leverage the full spectrum of annotator perspectives and significantly enhance classification performance in subjective NLP tasks compared to majority voting and individual annotator modeling. Regarding the aggregation approach, the multi-label and multitask approaches are better for modeling clustered annotators than an ensemble and model majority vote. The dataset is publicly available in GitHub: https://github.com/Tadesse-Destaw/Beyond-Majority-Voting.

Zitieren

APA

Belay, T. D., Ahmad, I. S., Abdulmumin, I., Ayele, A. A., Gelbukh, A., Ricárdez-Vázquez, E., Kolesnikova, O., Muhammad, S. H., & Yimam, S. M. (2026). Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks. Transactions of the Association for Computational Linguistics, 14, 1329-1347. https://doi.org/10.1162/tacl.a.731

Vancouver

Belay TD, Ahmad IS, Abdulmumin I, Ayele AA, Gelbukh A, Ricárdez-Vázquez E, et al. Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks. Transactions of the Association for Computational Linguistics. 2026;14:1329-1347. doi:10.1162/tacl.a.731

BibTeX
@article{Belay2026,
  author = {Belay, Tadesse Destaw and Ahmad, Ibrahim Said and Abdulmumin, Idris and Ayele, Abinew Ali and Gelbukh, Alexander and Ricárdez-Vázquez, Eusebio and Kolesnikova, Olga and Muhammad, Shamsuddeen Hassan and Yimam, Seid Muhie},
  title = {Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  volume = {14},
  pages = {1329--1347},
  doi = {10.1162/tacl.a.731}
}
Kostenlos starten →