Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model
Cognitive Science · 2024
DOI: 10.1111/cogs.13448 ↗At a glance
Journal article in Cognitive Science, 2024. Written by Eva Portelance and 2 co-authors. Institution: Department of Linguistics, McGill University. Major MEDLINE topics: Language, Learning. Cited 3 times so far (Crossref); 102 references. The full text is freely available (open access, CC BY 4.0). Funded by: Social Sciences and Humanities Research Council.
Abstract (original language)
Interpreting a seemingly simple function word like “or,” “behind,” or “more” can require logical, numerical, and relational reasoning. How are such words learned by children? Prior acquisition theories have often relied on positing a foundation of innate knowledge. Yet recent neural‐network‐based visual question answering models apparently can learn to use function words as part of answering questions about complex visual scenes. In this paper, we study what these models learn about function words, in the hope of better understanding how the meanings of these words can be learned by both models and children. We show that recurrent models trained on visually grounded language learn gradient semantics for function words requiring spatial and numerical reasoning. Furthermore, we find that these models can learn the meanings of logical connectives and and or without any prior knowledge of logical reasoning as well as early evidence that they are sensitive to alternative expressions when interpreting language. Finally, we show that word learning difficulty is dependent on the frequency of models' input. Our findings offer proof‐of‐concept evidence that it is possible to learn the nuanced interpretations of function words in a visually grounded context by using non‐symbolic general statistical learning algorithms, without any prior knowledge of linguistic meaning.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Linguistics, McGill University.
- Mila - Quebec Artificial Intelligence Institute.
- Department of Psychology, Stanford University.
- Department of Linguistics, Stanford University.
ORCID profiles: Eva Portelance, Michael C Frank, Dan Jurafsky
Funding
- Social Sciences and Humanities Research Council
Cite
APA
Portelance, E., Frank, M. C., & Jurafsky, D. (2024). Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model. Cognitive Science. https://doi.org/10.1111/cogs.13448
Vancouver
Portelance E, Frank MC, Jurafsky D. Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model. Cognitive Science. 2024. doi:10.1111/cogs.13448
BibTeX
@article{Portelance2024,
author = {Portelance, Eva and Frank, Michael C and Jurafsky, Dan},
title = {Learning the Meanings of Function Words From Grounded Language Using a Visual Question Answering Model},
journal = {Cognitive Science},
year = {2024},
doi = {10.1111/cogs.13448}
}
EUVIMED