EUVIMEDEUVIMED entdecken
Crossref

Holistic Evaluation of Language Models

Rishi Bommasani, Percy Liang, Tony Lee

Annals of the New York Academy of Sciences · 2023-07 · 1525 (1) · 140-146

DOI: 10.1111/nyas.15007 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Annals of the New York Academy of Sciences, online erschienen am 25.05.2023. Verfasst von Rishi Bommasani und 2 weiteren Autor:innen. Beteiligte Einrichtung: Center for Research on Foundation Models, Stanford University, Stanford, California, USA. Hauptthemen laut MEDLINE: Dietary Supplements, Knowledge. Die Arbeit wurde bisher 251-mal zitiert (Crossref) und nennt 65 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Language models (LMs) like GPT‐3, PaLM, and ChatGPT are the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of LMs. LMs can serve many purposes and their behavior should satisfy many desiderata. To navigate the vast space of potential scenarios and metrics, we taxonomize the space and select representative subsets. We evaluate models on 16 core scenarios and 7 metrics, exposing important trade‐offs. We supplement our core evaluation with seven targeted evaluations to deeply analyze specific aspects (including world knowledge, reasoning, regurgitation of copyrighted content, and generation of disinformation). We benchmark 30 LMs, from OpenAI, Microsoft, Google, Meta, Cohere, AI21 Labs, and others. Prior to HELM, models were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: all 30 models are now benchmarked under the same standardized conditions. Our evaluation surfaces 25 top‐level findings. For full transparency, we release all raw model prompts and completions publicly. HELM is a living benchmark for the community, continuously updated with new scenarios, metrics, and models https://crfm.stanford.edu/helm/latest/ .

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Dietary SupplementsKnowledgeHumansLanguageProblem SolvingTechnology

Stichwörter der Autor:innen

artificial intelligenceevaluationfoundation modelslanguage modelsnatural language processingtransparency

Autor:innen und Institutionen

  • Center for Research on Foundation Models, Stanford University, Stanford, California, USA.

Zitieren

APA

Bommasani, R., Liang, P., & Lee, T. (2023). Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences, 1525(1), 140-146. https://doi.org/10.1111/nyas.15007

Vancouver

Bommasani R, Liang P, Lee T. Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences. 2023;1525(1):140-146. doi:10.1111/nyas.15007

BibTeX
@article{Bommasani2023,
  author = {Bommasani, Rishi and Liang, Percy and Lee, Tony},
  title = {Holistic Evaluation of Language Models},
  journal = {Annals of the New York Academy of Sciences},
  year = {2023},
  volume = {1525},
  number = {1},
  pages = {140--146},
  doi = {10.1111/nyas.15007}
}
Kostenlos starten →