State of What Art? A Call for Multi-Prompt LLM Evaluation
Transactions of the Association for Computational Linguistics · 01.08.2024 · 12 · 933-949
DOI: 10.1162/tacl_a_00681 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.08.2024. Verfasst von Moran Mizrahi und 5 weiteren Autor:innen. Die Arbeit wurde bisher 118-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Recent advances in LLMs have led to an abundance of evaluation benchmarks, which typically rely on a single instruction template per task. We create a large-scale collection of instruction paraphrases and comprehensively analyze the brittleness introduced by single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. We find that different instruction templates lead to very different performance, both absolute and relative. Instead, we propose a set of diverse metrics on multiple instruction paraphrases, specifically tailored for different use cases (e.g., LLM vs. downstream development), ensuring a more reliable and meaningful assessment of LLM capabilities. We show that our metrics provide new insights into the strengths and limitations of current LLMs.
Zitieren
APA
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024). State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics, 12, 933-949. https://doi.org/10.1162/tacl_a_00681
Vancouver
Mizrahi M, Kaplan G, Malkin D, Dror R, Shahaf D, Stanovsky G. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics. 2024;12:933-949. doi:10.1162/tacl_a_00681
BibTeX
@article{Mizrahi2024,
author = {Mizrahi, Moran and Kaplan, Guy and Malkin, Dan and Dror, Rotem and Shahaf, Dafna and Stanovsky, Gabriel},
title = {State of What Art? A Call for Multi-Prompt LLM Evaluation},
journal = {Transactions of the Association for Computational Linguistics},
year = {2024},
volume = {12},
pages = {933--949},
doi = {10.1162/tacl_a_00681}
}
EUVIMED