EUVIMEDSkopri EUVIMED
Open access · CC BY 4.0

Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model

Eva Portelance iD, Michael C Frank iD, Dan Jurafsky iD

Cognitive Science · 2024

DOI: 10.1111/cogs.13448 ↗

At a glance

Journal article in Cognitive Science, 2024. Written by Eva Portelance and 2 co-authors. Institution: Department of Linguistics, McGill University. Major MEDLINE topics: Language, Learning. Cited 3 times so far (Crossref); 102 references. The full text is freely available (open access, CC BY 4.0). Funded by: Social Sciences and Humanities Research Council.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Interpreting a seemingly simple function word like “or,” “behind,” or “more” can require logical, numerical, and relational reasoning. How are such words learned by children? Prior acquisition theories have often relied on positing a foundation of innate knowledge. Yet recent neural‐network‐based visual question answering models apparently can learn to use function words as part of answering questions about complex visual scenes. In this paper, we study what these models learn about function words, in the hope of better understanding how the meanings of these words can be learned by both models and children. We show that recurrent models trained on visually grounded language learn gradient semantics for function words requiring spatial and numerical reasoning. Furthermore, we find that these models can learn the meanings of logical connectives and and or without any prior knowledge of logical reasoning as well as early evidence that they are sensitive to alternative expressions when interpreting language. Finally, we show that word learning difficulty is dependent on the frequency of models' input. Our findings offer proof‐of‐concept evidence that it is possible to learn the nuanced interpretations of function words in a visually grounded context by using non‐symbolic general statistical learning algorithms, without any prior knowledge of linguistic meaning.

Subjects

MeSH terms (MEDLINE)

LanguageLearningHumansSemanticsLanguage DevelopmentNeural Networks, ComputerChildLogic

Author keywords

Function word acquisitionLogical reasoningMultimodal statistical learningNeural network modelsProof of conceptReasoning skillsVisual question answeringVisually‐grounded language

Authors and institutions

  • Department of Linguistics, McGill University.
  • Mila - Quebec Artificial Intelligence Institute.
  • Department of Psychology, Stanford University.
  • Department of Linguistics, Stanford University.

ORCID profiles: Eva Portelance, Michael C Frank, Dan Jurafsky

Funding

  • Social Sciences and Humanities Research Council

Cite

APA

Portelance, E., Frank, M. C., & Jurafsky, D. (2024). Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model. Cognitive Science. https://doi.org/10.1111/cogs.13448

Vancouver

Portelance E, Frank MC, Jurafsky D. Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model. Cognitive Science. 2024. doi:10.1111/cogs.13448

BibTeX
@article{Portelance2024,
  author = {Portelance, Eva and Frank, Michael C and Jurafsky, Dan},
  title = {Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model},
  journal = {Cognitive Science},
  year = {2024},
  doi = {10.1111/cogs.13448}
}
Start for free →