EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

LLM-as-a-Judge: automated evaluation of search query parsing using large language models

Mehmet Selman Baysan, Serkan Uysal, İrem İşlek, Çağla Çığ Karaman, Tunga Güngör

Frontiers in Big Data · 21.07.2025 · 8

DOI: 10.3389/fdata.2025.1611389 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Big Data, online erschienen am 21.07.2025. Verfasst von Mehmet Selman Baysan und 4 weiteren Autor:innen. Beteiligte Einrichtung: sahibinden.com, Istanbul, Türkiye. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

The adoption of Large Language Models (LLMs) in search systems necessitates new evaluation methodologies beyond traditional rule-based or manual approaches.

Methods

We propose a general framework for evaluating structured outputs using LLMs, focusing on search query parsing within an online classified platform. Our approach leverages LLMs' contextual reasoning capabilities through three evaluation methodologies: Pointwise, Pairwise, and Pass/Fail assessments. Additionally, we introduce a Contextual Evaluation Prompt Routing strategy to improve reliability and reduce hallucinations.

Results

Experiments conducted on both small- and large-scale datasets demonstrate that LLM-based evaluation achieves approximately 90% agreement with human judgments.

Discussion

These results validate LLM-driven evaluation as a scalable, interpretable, and effective alternative to traditional evaluation methods, providing robust query parsing for real-world search systems.

Schlagwörter

Stichwörter der Autor:innen

LLM-as-a-Judgeautomatic evaluationevaluation frameworkgenerative searchlarge language modelsquery understandingsearch query parsingstructured output evaluation

Autor:innen und Institutionen

  • sahibinden.com, Istanbul, Türkiye.
  • sahibinden.com, Ankara, Türkiye.
  • Department of Computer Engineering, Faculty of Engineering, Boğaziçi University, Istanbul, Türkiye.

Zitieren

APA

Baysan, M. S., Uysal, S., İşlek, İ., Çığ Karaman, Ç., & Güngör, T. (2025). LLM-as-a-Judge: automated evaluation of search query parsing using large language models. Frontiers in Big Data, 8. https://doi.org/10.3389/fdata.2025.1611389

Vancouver

Baysan MS, Uysal S, İşlek İ, Çığ Karaman Ç, Güngör T. LLM-as-a-Judge: automated evaluation of search query parsing using large language models. Frontiers in Big Data. 2025;8. doi:10.3389/fdata.2025.1611389

BibTeX
@article{Baysan2025,
  author = {Baysan, Mehmet Selman and Uysal, Serkan and İşlek, İrem and Çığ Karaman, Çağla and Güngör, Tunga},
  title = {LLM-as-a-Judge: automated evaluation of search query parsing using large language models},
  journal = {Frontiers in Big Data},
  year = {2025},
  volume = {8},
  doi = {10.3389/fdata.2025.1611389}
}
Kostenlos starten →