Don't Use LLMs to Make Relevance Judgments
Information Retrieval Research · 25.03.2025 · 1 (1) · 29-46
DOI: 10.54195/irrj.19625 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information Retrieval Research, online erschienen am 25.03.2025. Verfasst von Ian Soboroff. Beteiligte Einrichtung: National Institute of Standards and Technology, Gaithersburg, Maryland, USA. Die Arbeit wurde bisher 20-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Intramural NIST DOC.
Abstract (Originalsprache)
Relevance judgments and other truth data for information retrieval (IR) evaluations are created manually. There is a strong temptation to use large language models (LLMs) as proxies for human judges. However, letting the LLM write your truth data handicaps the evaluation by setting that LLM as a ceiling on performance. There are ways to use LLMs in the relevance assessment process, but just generating relevance judgments with a prompt isn’t one of them.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- National Institute of Standards and Technology, Gaithersburg, Maryland, USA.
Förderung
- Intramural NIST DOC
Zitieren
APA
Soboroff, I. (2025). Don't Use LLMs to Make Relevance Judgments. Information Retrieval Research, 1(1), 29-46. https://doi.org/10.54195/irrj.19625
Vancouver
Soboroff I. Don't Use LLMs to Make Relevance Judgments. Information Retrieval Research. 2025;1(1):29-46. doi:10.54195/irrj.19625
BibTeX
@article{Soboroff2025,
author = {Soboroff, Ian},
title = {Don't Use LLMs to Make Relevance Judgments},
journal = {Information Retrieval Research},
year = {2025},
volume = {1},
number = {1},
pages = {29--46},
doi = {10.54195/irrj.19625}
}
EUVIMED