Evaluating the language abilities of Large Language Models vs. humans: Three caveats
Biolinguistics · 19.04.2024 · 18
DOI: 10.5964/bioling.14391 ↗Kurz gefasst
Fachartikel in der Zeitschrift Biolinguistics, online erschienen am 19.04.2024. Verfasst von Evelina Leivada und 2 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 14 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
We identify and analyze three caveats that may arise when analyzing the linguistic abilities of Large Language Models. The problem of unlicensed generalizations refers to the danger of interpreting performance in one task as predictive of the models’ overall capabilities, based on the assumption that because a specific task performance is indicative of certain underlying capabilities in humans, the same association holds for models. The human-like paradox refers to the problem of lacking human comparisons, while at the same time attributing human-like abilities to the models. Last, the problem of double standards refers to the use of tasks and methodologies that either cannot be applied to humans or they are evaluated differently in models vs. humans. While we recognize the impressive linguistic abilities of LLMs, we conclude that specific claims about the models’ human-likeness in the grammatical domain are premature.
Zitieren
APA
Leivada, E., Dentella, V., & Günther, F. (2024). Evaluating the language abilities of Large Language Models vs. humans: Three caveats. Biolinguistics, 18. https://doi.org/10.5964/bioling.14391
Vancouver
Leivada E, Dentella V, Günther F. Evaluating the language abilities of Large Language Models vs. humans: Three caveats. Biolinguistics. 2024;18. doi:10.5964/bioling.14391
BibTeX
@article{Leivada2024,
author = {Leivada, Evelina and Dentella, Vittoria and Günther, Fritz},
title = {Evaluating the language abilities of Large Language Models vs. humans: Three caveats},
journal = {Biolinguistics},
year = {2024},
volume = {18},
doi = {10.5964/bioling.14391}
}
EUVIMED