EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training

Eghbal A Hosseini iD, Martin Schrimpf iD, Yian Zhang, Samuel Bowman iD, Noga Zaslavsky iD, Evelina Fedorenko iD

Neurobiology of Language · 01.04.2024 · 5 (1) · 43-63

DOI: 10.1162/nol_a_00137 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Neurobiology of Language, online erschienen am 01.04.2024. Verfasst von Eghbal A Hosseini und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 84 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDCD NIH HHS, NINDS NIH HHS, National Institute of Neurological Disorders and Stroke.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models’ ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity—a measure of next-word prediction performance—is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models’ predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.

Schlagwörter

Stichwörter der Autor:innen

ANN-neural data alignmentartificial neural networkdevelopmenthuman behaviorlanguage network

Autor:innen und Institutionen

  • Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
  • McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
  • The MIT Quest for Intelligence Initiative, Cambridge, MA, USA.
  • Swiss Federal Institute of Technology, Lausanne, Switzerland.
  • Computer Science Department, Stanford University, Stanford, CA, USA.
  • Center for Data Science, New York University, New York, NY, USA.

ORCID-Profile: Eghbal A Hosseini, Martin Schrimpf, Samuel Bowman, Noga Zaslavsky, Evelina Fedorenko

Förderung

  • NIDCD NIH HHS
  • NINDS NIH HHS
  • National Institute of Neurological Disorders and Stroke
  • McGovern Institute for Brain Research, Massachusetts Institute of Technology
  • National Institute on Deafness and Other Communication Disorders
  • Simons Center for the Social Brain, Massachusetts Institute of Technology
  • Massachusetts Institute of Technology

Zitieren

APA

Hosseini, E. A., Schrimpf, M., Zhang, Y., Bowman, S., Zaslavsky, N., & Fedorenko, E. (2024). Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training. Neurobiology of Language, 5(1), 43-63. https://doi.org/10.1162/nol_a_00137

Vancouver

Hosseini EA, Schrimpf M, Zhang Y, Bowman S, Zaslavsky N, Fedorenko E. Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training. Neurobiology of Language. 2024;5(1):43-63. doi:10.1162/nol_a_00137

BibTeX
@article{Hosseini2024,
  author = {Hosseini, Eghbal A and Schrimpf, Martin and Zhang, Yian and Bowman, Samuel and Zaslavsky, Noga and Fedorenko, Evelina},
  title = {Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training},
  journal = {Neurobiology of Language},
  year = {2024},
  volume = {5},
  number = {1},
  pages = {43--63},
  doi = {10.1162/nol_a_00137}
}
Kostenlos starten →