EUVIMEDEUVIMED entdecken
Crossref

A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators

Chen Zhang, Luis Fernando D'Haro, Yiming Chen, Malu Zhang, Haizhou Li

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (17) · 19515-19524

DOI: 10.1609/aaai.v38i17.29923 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Chen Zhang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Automatic evaluation is an integral aspect of dialogue system research. The traditional reference-based NLG metrics are generally found to be unsuitable for dialogue assessment. Consequently, recent studies have suggested various unique, reference-free neural metrics that better align with human evaluations. Notably among them, large language models (LLMs), particularly the instruction-tuned variants like ChatGPT, are shown to be promising substitutes for human judges. Yet, existing works on utilizing LLMs for automatic dialogue evaluation are limited in their scope in terms of the number of meta-evaluation datasets, mode of evaluation, coverage of LLMs, etc. Hence, it remains inconclusive how effective these LLMs are. To this end, we conduct a comprehensive study on the application of LLMs for automatic dialogue evaluation. Specifically, we analyze the multi-dimensional evaluation capability of 30 recently emerged LLMs at both turn and dialogue levels, using a comprehensive set of 12 meta-evaluation datasets. Additionally, we probe the robustness of the LLMs in handling various adversarial perturbations at both turn and dialogue levels. Finally, we explore how model-level and dimension-level ensembles impact the evaluation performance. All resources are available at https://github.com/e0397123/comp-analysis.

Zitieren

APA

Zhang, C., D'Haro, L. F., Chen, Y., Zhang, M., & Li, H. (2024). A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17), 19515-19524. https://doi.org/10.1609/aaai.v38i17.29923

Vancouver

Zhang C, D'Haro LF, Chen Y, Zhang M, Li H. A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(17):19515-19524. doi:10.1609/aaai.v38i17.29923

BibTeX
@article{Zhang2024,
  author = {Zhang, Chen and D'Haro, Luis Fernando and Chen, Yiming and Zhang, Malu and Li, Haizhou},
  title = {A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {17},
  pages = {19515--19524},
  doi = {10.1609/aaai.v38i17.29923}
}
Kostenlos starten →