Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
Applied Sciences · 07.05.2025 · 15 (9) · 5198
DOI: 10.3390/app15095198 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 07.05.2025. Verfasst von Jaekeol Choi. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hankuk University of Foreign Studies.
Abstract (Originalsprache)
Evaluating query-passage relevance is a crucial task in information retrieval (IR), where the performance of large language models (LLMs) greatly depends on the quality of prompts. Current prompt optimization methods typically require multiple candidate generations or iterative refinements, resulting in significant computational overhead and limited practical applicability. In this paper, we propose a novel prompt optimization method that leverages LLM-based confusion matrix feedback to efficiently optimize prompts for the relevance evaluation task. Unlike previous approaches, our method systematically analyzes LLM predictions—both correct and incorrect—using a confusion matrix, enabling prompt refinement through a single-step update. Our experiments in realistic IR scenarios demonstrate that our method achieves competitive or superior performance compared to existing methods while drastically reducing computational costs, highlighting its potential as a practical and scalable solution.
Förderung
- Hankuk University of Foreign Studies
Zitieren
APA
Choi, J. (2025). Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback. Applied Sciences, 15(9), 5198. https://doi.org/10.3390/app15095198
Vancouver
Choi J. Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback. Applied Sciences. 2025;15(9):5198. doi:10.3390/app15095198
BibTeX
@article{Choi2025,
author = {Choi, Jaekeol},
title = {Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback},
journal = {Applied Sciences},
year = {2025},
volume = {15},
number = {9},
pages = {5198},
doi = {10.3390/app15095198}
}
EUVIMED