EUVIMEDEUVIMED entdecken
Crossref

On the Use of LLMs for Relevance Labelling

Marwah Alaofi, Paul Thomas, Falk Scholer, Mark Sanderson

ACM Transactions on Information Systems · 31.05.2026 · 44 (4) · 1-31

DOI: 10.1145/3788872 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 17.04.2026. Verfasst von Marwah Alaofi und 3 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: Australian Research Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) are increasingly used to replace human judges to assess the relevance of information objects, raising concerns about circularity, bias, and whether simulated preferences can substitute for human judgement. This work presents experiments using multiple LLMs to label passages for relevance. It examines their gullibility—how easily they are misled into labelling irrelevant passages as relevant. It also compares LLMs with human judges in ranking systems, analysing differences in discriminative power and whether some systems benefit under LLM-based evaluation. Results show that LLMs are influenced by the presence of query terms, even with irrelevant or random passages. Moreover, LLM-generated rankings are highly correlated with those of human judges, with strong agreement on which system is better in pairwise comparisons. However, LLMs may exhibit lower discriminative power, as seen in flatter ranking slopes and missed significance for meaningful improvements. Yet, there are no cases where capable LLMs and human judges reach opposing conclusions with significance. LLMs may boost traditional systems more than neural ones, adding a new concern of system bias. These findings highlight the strong potential of LLMs for relevance labelling, while also highlighting failure cases that call for careful adoption and further research to maintain evaluation integrity. 1

Förderung

  • Australian Research Council

Zitieren

APA

Alaofi, M., Thomas, P., Scholer, F., & Sanderson, M. (2026). On the Use of LLMs for Relevance Labelling. ACM Transactions on Information Systems, 44(4), 1-31. https://doi.org/10.1145/3788872

Vancouver

Alaofi M, Thomas P, Scholer F, Sanderson M. On the Use of LLMs for Relevance Labelling. ACM Transactions on Information Systems. 2026;44(4):1-31. doi:10.1145/3788872

BibTeX
@article{Alaofi2026,
  author = {Alaofi, Marwah and Thomas, Paul and Scholer, Falk and Sanderson, Mark},
  title = {On the Use of LLMs for Relevance Labelling},
  journal = {ACM Transactions on Information Systems},
  year = {2026},
  volume = {44},
  number = {4},
  pages = {1--31},
  doi = {10.1145/3788872}
}
Kostenlos starten →