TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer
IEEE Transactions on Pattern Analysis and Machine Intelligence · 01.11.2023 · 45 (11) · 13636-13652
DOI: 10.1109/tpami.2023.3296823 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, erschienen am 01.11.2023. Verfasst von Jiajun Deng und 7 weiteren Autor:innen. Die Arbeit wurde bisher 95-mal zitiert (Crossref) und nennt 94 Literaturangaben. Gefördert von: National Key RD Program of China, National Natural Science Foundation of China, Shanghai Committee of Science and Technology.
Abstract (Originalsprache)
In this work, we explore neat yet effective Transformer-based frameworks for visual grounding. The previous methods generally address the core problem of visual grounding, i.e., multi-modal fusion and reasoning, with manually-designed mechanisms. Such heuristic designs are not only complicated but also make models easily overfit specific data distributions. To avoid this, we first propose TransVG, which establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates. We empirically show that complicated fusion modules can be replaced by a simple stack of Transformer encoder layers with higher performance. However, the core fusion Transformer in TransVG is stand-alone against uni-modal encoders, and thus should be trained from scratch on limited visual grounding data, which makes it hard to be optimized and leads to sub-optimal performance. To this end, we further introduce TransVG++ to make two-fold improvements. For one thing, we upgrade our framework to a purely Transformer-based one by leveraging Vision Transformer (ViT) for vision feature encoding. For another, we devise Language Conditioned Vision Transformer that removes external fusion modules and reuses the uni-modal ViT for vision-language fusion at the intermediate layers. We conduct extensive experiments on five prevalent datasets, and report a series of state-of-the-art records.
Förderung
- National Key RD Program of China
- National Natural Science Foundation of China
- Shanghai Committee of Science and Technology
- GPU
- Supercomputing Center of the USTC
Zitieren
APA
Deng, J., Yang, Z., Liu, D., Chen, T., Zhou, W., Zhang, Y., Li, H., & Ouyang, W. (2023). TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 13636-13652. https://doi.org/10.1109/tpami.2023.3296823
Vancouver
Deng J, Yang Z, Liu D, Chen T, Zhou W, Zhang Y, et al. TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(11):13636-13652. doi:10.1109/tpami.2023.3296823
BibTeX
@article{Deng2023,
author = {Deng, Jiajun and Yang, Zhengyuan and Liu, Daqing and Chen, Tianlang and Zhou, Wengang and Zhang, Yanyong and Li, Houqiang and Ouyang, Wanli},
title = {TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2023},
volume = {45},
number = {11},
pages = {13636--13652},
doi = {10.1109/tpami.2023.3296823}
}
EUVIMED