EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Don't Use LLMs to Make Relevance Judgments

Ian Soboroff

Information Retrieval Research · 25.03.2025 · 1 (1) · 29-46

DOI: 10.54195/irrj.19625 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information Retrieval Research, online erschienen am 25.03.2025. Verfasst von Ian Soboroff. Beteiligte Einrichtung: National Institute of Standards and Technology, Gaithersburg, Maryland, USA. Die Arbeit wurde bisher 20-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Intramural NIST DOC.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Relevance judgments and other truth data for information retrieval (IR) evaluations are created manually. There is a strong temptation to use large language models (LLMs) as proxies for human judges. However, letting the LLM write your truth data handicaps the evaluation by setting that LLM as a ceiling on performance. There are ways to use LLMs in the relevance assessment process, but just generating relevance judgments with a prompt isn’t one of them.

Schlagwörter

Stichwörter der Autor:innen

Information Retrieval EvaluationLarge Language Models

Autor:innen und Institutionen

  • National Institute of Standards and Technology, Gaithersburg, Maryland, USA.

Förderung

  • Intramural NIST DOC

Zitieren

APA

Soboroff, I. (2025). Don't Use LLMs to Make Relevance Judgments. Information Retrieval Research, 1(1), 29-46. https://doi.org/10.54195/irrj.19625

Vancouver

Soboroff I. Don't Use LLMs to Make Relevance Judgments. Information Retrieval Research. 2025;1(1):29-46. doi:10.54195/irrj.19625

BibTeX
@article{Soboroff2025,
  author = {Soboroff, Ian},
  title = {Don't Use LLMs to Make Relevance Judgments},
  journal = {Information Retrieval Research},
  year = {2025},
  volume = {1},
  number = {1},
  pages = {29--46},
  doi = {10.54195/irrj.19625}
}
Kostenlos starten →