Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training
Neurobiology of Language · 01.04.2024 · 5 (1) · 43-63
DOI: 10.1162/nol_a_00137 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neurobiology of Language, online erschienen am 01.04.2024. Verfasst von Eghbal A Hosseini und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 84 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDCD NIH HHS, NINDS NIH HHS, National Institute of Neurological Disorders and Stroke.
Abstract (Originalsprache)
Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models’ ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity—a measure of next-word prediction performance—is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models’ predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- The MIT Quest for Intelligence Initiative, Cambridge, MA, USA.
- Swiss Federal Institute of Technology, Lausanne, Switzerland.
- Computer Science Department, Stanford University, Stanford, CA, USA.
- Center for Data Science, New York University, New York, NY, USA.
ORCID-Profile: Eghbal A Hosseini, Martin Schrimpf, Samuel Bowman, Noga Zaslavsky, Evelina Fedorenko
Förderung
- NIDCD NIH HHS
- NINDS NIH HHS
- National Institute of Neurological Disorders and Stroke
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- National Institute on Deafness and Other Communication Disorders
- Simons Center for the Social Brain, Massachusetts Institute of Technology
- Massachusetts Institute of Technology
Zitieren
APA
Hosseini, E. A., Schrimpf, M., Zhang, Y., Bowman, S., Zaslavsky, N., & Fedorenko, E. (2024). Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training. Neurobiology of Language, 5(1), 43-63. https://doi.org/10.1162/nol_a_00137
Vancouver
Hosseini EA, Schrimpf M, Zhang Y, Bowman S, Zaslavsky N, Fedorenko E. Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training. Neurobiology of Language. 2024;5(1):43-63. doi:10.1162/nol_a_00137
BibTeX
@article{Hosseini2024,
author = {Hosseini, Eghbal A and Schrimpf, Martin and Zhang, Yian and Bowman, Samuel and Zaslavsky, Noga and Fedorenko, Evelina},
title = {Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training},
journal = {Neurobiology of Language},
year = {2024},
volume = {5},
number = {1},
pages = {43--63},
doi = {10.1162/nol_a_00137}
}
EUVIMED