EUVIMEDEUVIMED entdecken
Crossref

Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?

Muhammad Umair iD, Julia B Mertens iD, Lena Warnke, Jan P de Ruiter

Cognitive Science · 2024-11 · 48 (11)

DOI: 10.1111/cogs.70013 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Cognitive Science, online erschienen am 26.11.2024. Verfasst von Muhammad Umair und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Tufts University. Hauptthemen laut MEDLINE: Language, Speech. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 93 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformer‐based Large Language Models (LLMs) have recently increased in popularity, in part due to their impressive performance on a number of language tasks. While LLMs can produce human‐like writing, the extent to which these models can learn to predict spoken language in natural interaction remains unclear. This is a nontrivial question, as spoken and written language differ in syntax, pragmatics, and norms that interlocutors follow. Previous work suggests that while LLMs may develop an understanding of linguistic rules based on statistical regularities, they fail to acquire the knowledge required for language use. This implies that LLMs may not learn the normative structure underlying interactive spoken language, but may instead only model superficial regularities in speech. In this paper, we aim to evaluate LLMs as models of spoken dialogue. Specifically, we investigate whether LLMs can learn that the identity of a speaker in spoken dialogue influences what is likely to be said. To answer this question, we first fine‐tuned two variants of a specific LLM (GPT‐2) on transcripts of natural spoken dialogue in English. Then, we used these models to compute surprisal values for two‐turn sequences with the same first‐turn but different second‐turn speakers and compared the output to human behavioral data. While the predictability of words in all fine‐tuned models was influenced by speaker identity information, the models did not replicate humans' use of this information. Our findings suggest that although LLMs may learn to generate text conforming to normative linguistic structure, they do not (yet) faithfully replicate human behavior in natural conversation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguageSpeechHumansWritingLearningLinguistics

Stichwörter der Autor:innen

Generative pretrained transformersLanguage in interactionNatural language processing

Autor:innen und Institutionen

  • Department of Computer Science, Tufts University.
  • Department of Psychology, Tufts University.

ORCID-Profile: Muhammad Umair, Julia B Mertens

Zitieren

APA

Umair, M., Mertens, J. B., Warnke, L., & de Ruiter, J. P. (2024). Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?. Cognitive Science, 48(11). https://doi.org/10.1111/cogs.70013

Vancouver

Umair M, Mertens JB, Warnke L, de Ruiter JP. Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?. Cognitive Science. 2024;48(11). doi:10.1111/cogs.70013

BibTeX
@article{Umair2024,
  author = {Umair, Muhammad and Mertens, Julia B and Warnke, Lena and de Ruiter, Jan P},
  title = {Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?},
  journal = {Cognitive Science},
  year = {2024},
  volume = {48},
  number = {11},
  doi = {10.1111/cogs.70013}
}
Kostenlos starten →