EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely

Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, Alessandro Lenci

Cognitive Science · 2023-11 · 47 (11)

DOI: 10.1111/cogs.13386 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Cognitive Science, online erschienen am 27.11.2023. Verfasst von Carina Kauf und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology. Hauptthemen laut MEDLINE: Language, Semantics. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 172 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Commission.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Word co‐occurrence patterns in language corpora contain a surprising amount of conceptual knowledge. Large language models (LLMs), trained to predict words in context, leverage these patterns to achieve impressive performance on diverse semantic tasks requiring world knowledge. An important but understudied question about LLMs’ semantic abilities is whether they acquire generalized knowledge of common events. Here, we test whether five pretrained LLMs (from 2018's BERT to 2023's MPT) assign a higher likelihood to plausible descriptions of agent−patient interactions than to minimally different implausible versions of the same event. Using three curated sets of minimal sentence pairs (total n = 1215), we found that pretrained LLMs possess substantial event knowledge, outperforming other distributional language models. In particular, they almost always assign a higher likelihood to possible versus impossible events ( The teacher bought the laptop vs. The laptop bought the teacher ). However, LLMs show less consistent preferences for likely versus unlikely events ( The nanny tutored the boy vs. The boy tutored the nanny ). In follow‐up analyses, we show that (i) LLM scores are driven by both plausibility and surface‐level sentence features, (ii) LLM scores generalize well across syntactic variants (active vs. passive constructions) but less well across semantic variants (synonymous sentences), (iii) some LLM errors mirror human judgment ambiguity, and (iv) sentence plausibility serves as an organizing dimension in internal LLM representations. Overall, our results show that important aspects of event knowledge naturally emerge from distributional linguistic patterns, but also highlight a gap between representations of possible/impossible and likely/unlikely events.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguageSemanticsMaleHumansKnowledgeReadingJudgment

Stichwörter der Autor:innen

Artificial neural networksGeneralized event knowledgeLanguage modelsPlausibilitySemanticsSyntaxTypicalityWorld knowledge

Autor:innen und Institutionen

  • Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology.
  • McGovern Institute for Brain Research, Massachusetts Institute of Technology.
  • Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology.
  • Department of Modern Languages, Literatures and Cultures, University of Bologna.
  • Department of Chinese and Bilingual Studies, Hong Kong Polytechnic University.
  • Department of Mathematics, University of Washington.

Förderung

  • European Commission

Zitieren

APA

Kauf, C., Ivanova, A. A., Rambelli, G., Chersoni, E., She, J. S., Chowdhury, Z., Fedorenko, E., & Lenci, A. (2023). Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely. Cognitive Science, 47(11). https://doi.org/10.1111/cogs.13386

Vancouver

Kauf C, Ivanova AA, Rambelli G, Chersoni E, She JS, Chowdhury Z, et al. Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely. Cognitive Science. 2023;47(11). doi:10.1111/cogs.13386

BibTeX
@article{Kauf2023,
  author = {Kauf, Carina and Ivanova, Anna A and Rambelli, Giulia and Chersoni, Emmanuele and She, Jingyuan Selena and Chowdhury, Zawad and Fedorenko, Evelina and Lenci, Alessandro},
  title = {Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely},
  journal = {Cognitive Science},
  year = {2023},
  volume = {47},
  number = {11},
  doi = {10.1111/cogs.13386}
}
Kostenlos starten →