EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

UNCERTAINTY-AWARE LLM-AS-A-JUDGE EVALUATION FOR RELIABLE SEARCH AND RECOMMENDATION SYSTEMS: CALIBRATED CONFIDENCE AND IMPACT-AWARE SELECTIVE HUMAN REVIEW

GaiKe Wang, PengFei Li, EnMiao Feng, TianJun Mo

AI and Data Science Journal · 31.12.2024 · 1 (1) · 112-121

DOI: 10.61784/adsj4043 ↗

Kurz gefasst

Fachartikel in der Zeitschrift AI and Data Science Journal, online erschienen am 31.12.2024. Verfasst von GaiKe Wang und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are increasingly used as judges that produce relevance labels for evaluating search and recommendation systems, yet their graded judgments fluctuate with the prompt, the model and the assessment criterion, disagree with human assessors in a topic-dependent way, and come without trustworthy confidence. This paper presents CARE (Consistency-Aware calibration with Risk- and Evaluation-impact-driven review), an uncertainty-aware framework that turns a panel of LLM judgments into calibrated relevance distributions and spends a limited human budget where it matters most for evaluation outcomes. CARE encodes multi-dimensional consistency signals across repeated runs, prompts, LLMs, criteria and fine-tuning; calibrates them with a hierarchical multinomial model whose topic-specific grade biases receive an empirical-Bayes prior and a Laplace posterior; and routes pairs to assessors by expected ordinal error weighted by their impact on system comparisons, complemented by online posterior updates, soft expected-gain qrels and risk-targeted auto-acceptance. On the TREC DL 2023-based LLMJudge benchmark with 33 real LLM label sets (GPT-4o, Llama-3-8B/70B and FLAN-T5 judges using zero-shot, chain-of-thought, few-shot, criteria-decomposed and fine-tuned pipelines), changing only the prompt flips 22–46% of graded labels, whereas repeated sampling flips fewer than 0.3%. With ten human labels per topic, CARE raises Cohen’s κ with TREC assessors from 0.273 (majority vote) to 0.412 and reduces the expected calibration error from 0.150 to 0.036. Reviewing 10% of the pairs selected by CARE lifts the Kendall’s τ between system rankings from 0.642 (random review) to 0.895, and τ≥0.85 is reached with 7.7% of the judging effort.

Zitieren

APA

Wang, G., Li, P., Feng, E., & Mo, T. (2024). UNCERTAINTY-AWARE LLM-AS-A-JUDGE EVALUATION FOR RELIABLE SEARCH AND RECOMMENDATION SYSTEMS: CALIBRATED CONFIDENCE AND IMPACT-AWARE SELECTIVE HUMAN REVIEW. AI and Data Science Journal, 1(1), 112-121. https://doi.org/10.61784/adsj4043

Vancouver

Wang G, Li P, Feng E, Mo T. UNCERTAINTY-AWARE LLM-AS-A-JUDGE EVALUATION FOR RELIABLE SEARCH AND RECOMMENDATION SYSTEMS: CALIBRATED CONFIDENCE AND IMPACT-AWARE SELECTIVE HUMAN REVIEW. AI and Data Science Journal. 2024;1(1):112-121. doi:10.61784/adsj4043

BibTeX
@article{Wang2024,
  author = {Wang, GaiKe and Li, PengFei and Feng, EnMiao and Mo, TianJun},
  title = {UNCERTAINTY-AWARE LLM-AS-A-JUDGE EVALUATION FOR RELIABLE SEARCH AND RECOMMENDATION SYSTEMS: CALIBRATED CONFIDENCE AND IMPACT-AWARE SELECTIVE HUMAN REVIEW},
  journal = {AI and Data Science Journal},
  year = {2024},
  volume = {1},
  number = {1},
  pages = {112--121},
  doi = {10.61784/adsj4043}
}
Kostenlos starten →