Holistic Evaluation of Language Models
Annals of the New York Academy of Sciences · 2023-07 · 1525 (1) · 140-146
DOI: 10.1111/nyas.15007 ↗Kurz gefasst
Fachartikel in der Zeitschrift Annals of the New York Academy of Sciences, online erschienen am 25.05.2023. Verfasst von Rishi Bommasani und 2 weiteren Autor:innen. Beteiligte Einrichtung: Center for Research on Foundation Models, Stanford University, Stanford, California, USA. Hauptthemen laut MEDLINE: Dietary Supplements, Knowledge. Die Arbeit wurde bisher 251-mal zitiert (Crossref) und nennt 65 Literaturangaben.
Abstract (Originalsprache)
Language models (LMs) like GPT‐3, PaLM, and ChatGPT are the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of LMs. LMs can serve many purposes and their behavior should satisfy many desiderata. To navigate the vast space of potential scenarios and metrics, we taxonomize the space and select representative subsets. We evaluate models on 16 core scenarios and 7 metrics, exposing important trade‐offs. We supplement our core evaluation with seven targeted evaluations to deeply analyze specific aspects (including world knowledge, reasoning, regurgitation of copyrighted content, and generation of disinformation). We benchmark 30 LMs, from OpenAI, Microsoft, Google, Meta, Cohere, AI21 Labs, and others. Prior to HELM, models were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: all 30 models are now benchmarked under the same standardized conditions. Our evaluation surfaces 25 top‐level findings. For full transparency, we release all raw model prompts and completions publicly. HELM is a living benchmark for the community, continuously updated with new scenarios, metrics, and models https://crfm.stanford.edu/helm/latest/ .
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Research on Foundation Models, Stanford University, Stanford, California, USA.
Zitieren
APA
Bommasani, R., Liang, P., & Lee, T. (2023). Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences, 1525(1), 140-146. https://doi.org/10.1111/nyas.15007
Vancouver
Bommasani R, Liang P, Lee T. Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences. 2023;1525(1):140-146. doi:10.1111/nyas.15007
BibTeX
@article{Bommasani2023,
author = {Bommasani, Rishi and Liang, Percy and Lee, Tony},
title = {Holistic Evaluation of Language Models},
journal = {Annals of the New York Academy of Sciences},
year = {2023},
volume = {1525},
number = {1},
pages = {140--146},
doi = {10.1111/nyas.15007}
}
EUVIMED