LLM-as-a-Judge: automated evaluation of search query parsing using large language models
Frontiers in Big Data · 21.07.2025 · 8
DOI: 10.3389/fdata.2025.1611389 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Big Data, online erschienen am 21.07.2025. Verfasst von Mehmet Selman Baysan und 4 weiteren Autor:innen. Beteiligte Einrichtung: sahibinden.com, Istanbul, Türkiye. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Introduction
The adoption of Large Language Models (LLMs) in search systems necessitates new evaluation methodologies beyond traditional rule-based or manual approaches.
Methods
We propose a general framework for evaluating structured outputs using LLMs, focusing on search query parsing within an online classified platform. Our approach leverages LLMs' contextual reasoning capabilities through three evaluation methodologies: Pointwise, Pairwise, and Pass/Fail assessments. Additionally, we introduce a Contextual Evaluation Prompt Routing strategy to improve reliability and reduce hallucinations.
Results
Experiments conducted on both small- and large-scale datasets demonstrate that LLM-based evaluation achieves approximately 90% agreement with human judgments.
Discussion
These results validate LLM-driven evaluation as a scalable, interpretable, and effective alternative to traditional evaluation methods, providing robust query parsing for real-world search systems.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- sahibinden.com, Istanbul, Türkiye.
- sahibinden.com, Ankara, Türkiye.
- Department of Computer Engineering, Faculty of Engineering, Boğaziçi University, Istanbul, Türkiye.
Zitieren
APA
Baysan, M. S., Uysal, S., İşlek, İ., Çığ Karaman, Ç., & Güngör, T. (2025). LLM-as-a-Judge: automated evaluation of search query parsing using large language models. Frontiers in Big Data, 8. https://doi.org/10.3389/fdata.2025.1611389
Vancouver
Baysan MS, Uysal S, İşlek İ, Çığ Karaman Ç, Güngör T. LLM-as-a-Judge: automated evaluation of search query parsing using large language models. Frontiers in Big Data. 2025;8. doi:10.3389/fdata.2025.1611389
BibTeX
@article{Baysan2025,
author = {Baysan, Mehmet Selman and Uysal, Serkan and İşlek, İrem and Çığ Karaman, Çağla and Güngör, Tunga},
title = {LLM-as-a-Judge: automated evaluation of search query parsing using large language models},
journal = {Frontiers in Big Data},
year = {2025},
volume = {8},
doi = {10.3389/fdata.2025.1611389}
}
EUVIMED