EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback

Jaekeol Choi

Applied Sciences · 07.05.2025 · 15 (9) · 5198

DOI: 10.3390/app15095198 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 07.05.2025. Verfasst von Jaekeol Choi. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hankuk University of Foreign Studies.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Evaluating query-passage relevance is a crucial task in information retrieval (IR), where the performance of large language models (LLMs) greatly depends on the quality of prompts. Current prompt optimization methods typically require multiple candidate generations or iterative refinements, resulting in significant computational overhead and limited practical applicability. In this paper, we propose a novel prompt optimization method that leverages LLM-based confusion matrix feedback to efficiently optimize prompts for the relevance evaluation task. Unlike previous approaches, our method systematically analyzes LLM predictions—both correct and incorrect—using a confusion matrix, enabling prompt refinement through a single-step update. Our experiments in realistic IR scenarios demonstrate that our method achieves competitive or superior performance compared to existing methods while drastically reducing computational costs, highlighting its potential as a practical and scalable solution.

Förderung

  • Hankuk University of Foreign Studies

Zitieren

APA

Choi, J. (2025). Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback. Applied Sciences, 15(9), 5198. https://doi.org/10.3390/app15095198

Vancouver

Choi J. Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback. Applied Sciences. 2025;15(9):5198. doi:10.3390/app15095198

BibTeX
@article{Choi2025,
  author = {Choi, Jaekeol},
  title = {Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback},
  journal = {Applied Sciences},
  year = {2025},
  volume = {15},
  number = {9},
  pages = {5198},
  doi = {10.3390/app15095198}
}
Kostenlos starten →