GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (5) · 4766-4775
DOI: 10.1609/aaai.v38i5.28278 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Haozhan Shen und 3 weiteren Autor:innen. Die Arbeit wurde bisher 27-mal zitiert (Crossref).
Abstract (Originalsprache)
Visual grounding, a crucial vision-language task involving the understanding of the visual context based on the query expression, necessitates the model to capture the interactions between objects, as well as various spatial and attribute information. However, the annotation data of visual grounding task is limited due to its time-consuming and labor-intensive annotation process, resulting in the trained models being constrained from generalizing its capability to a broader domain. To address this challenge, we propose GroundVLP, a simple yet effective zero-shot method that harnesses visual grounding ability from the existing models trained from image-text pairs and pure object detection data, both of which are more conveniently obtainable and offer a broader domain compared to visual grounding annotation data. GroundVLP proposes a fusion mechanism that combines the heatmap from GradCAM and the object proposals of open-vocabulary detectors. We demonstrate that the proposed method significantly outperforms other zero-shot methods on RefCOCO/+/g datasets, surpassing prior zero-shot state-of-the-art by approximately 28% on the test split of RefCOCO and RefCOCO+. Furthermore, GroundVLP performs comparably to or even better than some non-VLP-based supervised models on the Flickr30k entities dataset. Our code is available at https://github.com/om-ai-lab/GroundVLP.
Zitieren
APA
Shen, H., Zhao, T., Zhu, M., & Yin, J. (2024). GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 38(5), 4766-4775. https://doi.org/10.1609/aaai.v38i5.28278
Vancouver
Shen H, Zhao T, Zhu M, Yin J. GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(5):4766-4775. doi:10.1609/aaai.v38i5.28278
BibTeX
@article{Shen2024,
author = {Shen, Haozhan and Zhao, Tiancheng and Zhu, Mingwei and Yin, Jianwei},
title = {GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {5},
pages = {4766--4775},
doi = {10.1609/aaai.v38i5.28278}
}
EUVIMED