Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks
Transactions of the Association for Computational Linguistics · 01.07.2026 · 14 · 1329-1347
DOI: 10.1162/tacl.a.731 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.07.2026. Verfasst von Tadesse Destaw Belay und 8 weiteren Autor:innen. Die Arbeit nennt 58 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has explored modeling individual annotators to preserve their perspectives. However, modeling each annotator is resource-intensive and remains underexplored across various NLP tasks. We propose an agreement-based clustering technique to model the disagreement between the annotators. We conduct comprehensive experiments in 40 datasets in 18 typologically diverse languages, covering three subjective NLP tasks: sentiment analysis, emotion classification, and hate speech detection. We evaluate four aggregation approaches: majority vote, ensemble, multi-label, and multitask. The results demonstrate that agreement-based clustering can leverage the full spectrum of annotator perspectives and significantly enhance classification performance in subjective NLP tasks compared to majority voting and individual annotator modeling. Regarding the aggregation approach, the multi-label and multitask approaches are better for modeling clustered annotators than an ensemble and model majority vote. The dataset is publicly available in GitHub: https://github.com/Tadesse-Destaw/Beyond-Majority-Voting.
Zitieren
APA
Belay, T. D., Ahmad, I. S., Abdulmumin, I., Ayele, A. A., Gelbukh, A., Ricárdez-Vázquez, E., Kolesnikova, O., Muhammad, S. H., & Yimam, S. M. (2026). Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks. Transactions of the Association for Computational Linguistics, 14, 1329-1347. https://doi.org/10.1162/tacl.a.731
Vancouver
Belay TD, Ahmad IS, Abdulmumin I, Ayele AA, Gelbukh A, Ricárdez-Vázquez E, et al. Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks. Transactions of the Association for Computational Linguistics. 2026;14:1329-1347. doi:10.1162/tacl.a.731
BibTeX
@article{Belay2026,
author = {Belay, Tadesse Destaw and Ahmad, Ibrahim Said and Abdulmumin, Idris and Ayele, Abinew Ali and Gelbukh, Alexander and Ricárdez-Vázquez, Eusebio and Kolesnikova, Olga and Muhammad, Shamsuddeen Hassan and Yimam, Seid Muhie},
title = {Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks},
journal = {Transactions of the Association for Computational Linguistics},
year = {2026},
volume = {14},
pages = {1329--1347},
doi = {10.1162/tacl.a.731}
}
EUVIMED