Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?
Cognitive Science · 2024-11 · 48 (11)
DOI: 10.1111/cogs.70013 ↗Kurz gefasst
Fachartikel in der Zeitschrift Cognitive Science, online erschienen am 26.11.2024. Verfasst von Muhammad Umair und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Tufts University. Hauptthemen laut MEDLINE: Language, Speech. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 93 Literaturangaben.
Abstract (Originalsprache)
Transformer‐based Large Language Models (LLMs) have recently increased in popularity, in part due to their impressive performance on a number of language tasks. While LLMs can produce human‐like writing, the extent to which these models can learn to predict spoken language in natural interaction remains unclear. This is a nontrivial question, as spoken and written language differ in syntax, pragmatics, and norms that interlocutors follow. Previous work suggests that while LLMs may develop an understanding of linguistic rules based on statistical regularities, they fail to acquire the knowledge required for language use. This implies that LLMs may not learn the normative structure underlying interactive spoken language, but may instead only model superficial regularities in speech. In this paper, we aim to evaluate LLMs as models of spoken dialogue. Specifically, we investigate whether LLMs can learn that the identity of a speaker in spoken dialogue influences what is likely to be said. To answer this question, we first fine‐tuned two variants of a specific LLM (GPT‐2) on transcripts of natural spoken dialogue in English. Then, we used these models to compute surprisal values for two‐turn sequences with the same first‐turn but different second‐turn speakers and compared the output to human behavioral data. While the predictability of words in all fine‐tuned models was influenced by speaker identity information, the models did not replicate humans' use of this information. Our findings suggest that although LLMs may learn to generate text conforming to normative linguistic structure, they do not (yet) faithfully replicate human behavior in natural conversation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, Tufts University.
- Department of Psychology, Tufts University.
ORCID-Profile: Muhammad Umair, Julia B Mertens
Zitieren
APA
Umair, M., Mertens, J. B., Warnke, L., & de Ruiter, J. P. (2024). Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?. Cognitive Science, 48(11). https://doi.org/10.1111/cogs.70013
Vancouver
Umair M, Mertens JB, Warnke L, de Ruiter JP. Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?. Cognitive Science. 2024;48(11). doi:10.1111/cogs.70013
BibTeX
@article{Umair2024,
author = {Umair, Muhammad and Mertens, Julia B and Warnke, Lena and de Ruiter, Jan P},
title = {Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?},
journal = {Cognitive Science},
year = {2024},
volume = {48},
number = {11},
doi = {10.1111/cogs.70013}
}
EUVIMED