Visual Spatial Reasoning
Transactions of the Association for Computational Linguistics · 20.06.2023 · 11 · 635-651
DOI: 10.1162/tacl_a_00566 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 20.06.2023. Verfasst von Fangyu Liu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 139-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Spatial relations are a basic part of human cognition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and-language models (VLMs) struggle to capture relational information. In this paper, we present Visual Spatial Reasoning (VSR), a dataset containing more than 10k natural text-image pairs with 66 types of spatial relations in English (e.g., under, in front of, facing). While using a seemingly simple annotation format, we show how the dataset includes challenging linguistic phenomena, such as varying reference frames. We demonstrate a large gap between human and model performance: The human ceiling is above 95%, while state-of-the-art models only achieve around 70%. We observe that VLMs’ by-relation performances have little correlation with the number of training examples and the tested models are in general incapable of recognising relations concerning the orientations of objects.1
Zitieren
APA
Liu, F., Emerson, G., & Collier, N. (2023). Visual Spatial Reasoning. Transactions of the Association for Computational Linguistics, 11, 635-651. https://doi.org/10.1162/tacl_a_00566
Vancouver
Liu F, Emerson G, Collier N. Visual Spatial Reasoning. Transactions of the Association for Computational Linguistics. 2023;11:635-651. doi:10.1162/tacl_a_00566
BibTeX
@article{Liu2023,
author = {Liu, Fangyu and Emerson, Guy and Collier, Nigel},
title = {Visual Spatial Reasoning},
journal = {Transactions of the Association for Computational Linguistics},
year = {2023},
volume = {11},
pages = {635--651},
doi = {10.1162/tacl_a_00566}
}
EUVIMED