EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication

Filippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt

Transactions of the Association for Computational Linguistics · 01.06.2026 · 14 · 1163-1185

DOI: 10.1162/tacl.a.701 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.06.2026. Verfasst von Filippo Merlo und 3 weiteren Autor:innen. Die Arbeit nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To what degree and under what conditions do VLMs rely on scene context when generating references to objects? To address this question, we introduce the Common Objects Out-of-Context (COOCo) dataset and conduct experiments on several VLMs under different degrees of scene–object congruency and noise. We find that models leverage scene context adaptively, depending on scene-object semantic relatedness and noise level. Based on these consistent trends across models, we turn to the question of how VLM attention patterns change as a function of target-scene semantic fit, and to what degree these patterns are predictive of categorisation accuracy. We find that successful object categorisation is associated with increased mid-layer attention to the target. We also find a non-monotonic dependency on semantic fit, with attention dropping at moderate fit and increasing for both low and high fit. These results suggest that VLMs dynamically balance local and contextual information for reference generation. Dataset and code are available here: https://github.com/cs-nlp-uu/scenereg.

Zitieren

APA

Merlo, F., Takmaz, E., Chen, W., & Gatt, A. (2026). Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication. Transactions of the Association for Computational Linguistics, 14, 1163-1185. https://doi.org/10.1162/tacl.a.701

Vancouver

Merlo F, Takmaz E, Chen W, Gatt A. Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication. Transactions of the Association for Computational Linguistics. 2026;14:1163-1185. doi:10.1162/tacl.a.701

BibTeX
@article{Merlo2026,
  author = {Merlo, Filippo and Takmaz, Ece and Chen, Wenkai and Gatt, Albert},
  title = {Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  volume = {14},
  pages = {1163--1185},
  doi = {10.1162/tacl.a.701}
}
Kostenlos starten →