Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (36) · 30235-30242
DOI: 10.1609/aaai.v40i36.40274 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Junjie Chen und 11 weiteren Autor:innen.
Abstract (Originalsprache)
The rapid development of large language models (LLMs) has highlighted the need for efficient and reliable methods to evaluate their performance. Traditional evaluation methods often face challenges like high costs, limited task formats, dependence on human references, and systematic biases. To address these limitations, we propose Auto-PRE, an automatic LLM evaluation framework inspired by the peer review process. Unlike previous approaches that rely on human annotations, Auto-PRE automatically selects evaluator LLMs based on three core traits: consistency, pertinence, and self-confidence, which correspond to the instruction, content, and response stages, respectively, and collectively cover the entire evaluation process. Experiments on three representative tasks, including summarization, non-factoid QA, and dialogue generation, demonstrate that Auto-PRE achieves state-of-the-art performance while significantly reducing evaluation costs. Furthermore, the structured and scalable design of our automatic qualification exam framework provides valuable insights into automating the evaluation of LLMs-as-judges, paving the way for more advanced LLM-based evaluation frameworks.
Zitieren
APA
Chen, J., Su, W., Chu, Z., Li, H., Zhou, Y., Yuan, D., Wang, X., Zhou, J., Liu, Y., Zhang, M., Ma, S., & Ai, Q. (2026). Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(36), 30235-30242. https://doi.org/10.1609/aaai.v40i36.40274
Vancouver
Chen J, Su W, Chu Z, Li H, Zhou Y, Yuan D, et al. Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(36):30235-30242. doi:10.1609/aaai.v40i36.40274
BibTeX
@article{Chen2026,
author = {Chen, Junjie and Su, Weihang and Chu, Zhumin and Li, Haitao and Zhou, Yujia and Yuan, Dingbo and Wang, Xudong and Zhou, Jun and Liu, Yiqun and Zhang, Min and Ma, Shaoping and Ai, Qingyao},
title = {Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {36},
pages = {30235--30242},
doi = {10.1609/aaai.v40i36.40274}
}
EUVIMED