EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

State of What Art? A Call for Multi-Prompt LLM Evaluation

Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, Gabriel Stanovsky

Transactions of the Association for Computational Linguistics · 01.08.2024 · 12 · 933-949

DOI: 10.1162/tacl_a_00681 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.08.2024. Verfasst von Moran Mizrahi und 5 weiteren Autor:innen. Die Arbeit wurde bisher 118-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advances in LLMs have led to an abundance of evaluation benchmarks, which typically rely on a single instruction template per task. We create a large-scale collection of instruction paraphrases and comprehensively analyze the brittleness introduced by single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. We find that different instruction templates lead to very different performance, both absolute and relative. Instead, we propose a set of diverse metrics on multiple instruction paraphrases, specifically tailored for different use cases (e.g., LLM vs. downstream development), ensuring a more reliable and meaningful assessment of LLM capabilities. We show that our metrics provide new insights into the strengths and limitations of current LLMs.

Zitieren

APA

Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024). State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics, 12, 933-949. https://doi.org/10.1162/tacl_a_00681

Vancouver

Mizrahi M, Kaplan G, Malkin D, Dror R, Shahaf D, Stanovsky G. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics. 2024;12:933-949. doi:10.1162/tacl_a_00681

BibTeX
@article{Mizrahi2024,
  author = {Mizrahi, Moran and Kaplan, Guy and Malkin, Dan and Dror, Rotem and Shahaf, Dafna and Stanovsky, Gabriel},
  title = {State of What Art? A Call for Multi-Prompt LLM Evaluation},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  volume = {12},
  pages = {933--949},
  doi = {10.1162/tacl_a_00681}
}
Kostenlos starten →