EUVIMEDEUVIMED entdecken
Crossref

TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer

Jiajun Deng, Zhengyuan Yang, Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang, Houqiang Li, Wanli Ouyang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 01.11.2023 · 45 (11) · 13636-13652

DOI: 10.1109/tpami.2023.3296823 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, erschienen am 01.11.2023. Verfasst von Jiajun Deng und 7 weiteren Autor:innen. Die Arbeit wurde bisher 95-mal zitiert (Crossref) und nennt 94 Literaturangaben. Gefördert von: National Key RD Program of China, National Natural Science Foundation of China, Shanghai Committee of Science and Technology.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this work, we explore neat yet effective Transformer-based frameworks for visual grounding. The previous methods generally address the core problem of visual grounding, i.e., multi-modal fusion and reasoning, with manually-designed mechanisms. Such heuristic designs are not only complicated but also make models easily overfit specific data distributions. To avoid this, we first propose TransVG, which establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates. We empirically show that complicated fusion modules can be replaced by a simple stack of Transformer encoder layers with higher performance. However, the core fusion Transformer in TransVG is stand-alone against uni-modal encoders, and thus should be trained from scratch on limited visual grounding data, which makes it hard to be optimized and leads to sub-optimal performance. To this end, we further introduce TransVG++ to make two-fold improvements. For one thing, we upgrade our framework to a purely Transformer-based one by leveraging Vision Transformer (ViT) for vision feature encoding. For another, we devise Language Conditioned Vision Transformer that removes external fusion modules and reuses the uni-modal ViT for vision-language fusion at the intermediate layers. We conduct extensive experiments on five prevalent datasets, and report a series of state-of-the-art records.

Förderung

  • National Key RD Program of China
  • National Natural Science Foundation of China
  • Shanghai Committee of Science and Technology
  • GPU
  • Supercomputing Center of the USTC

Zitieren

APA

Deng, J., Yang, Z., Liu, D., Chen, T., Zhou, W., Zhang, Y., Li, H., & Ouyang, W. (2023). TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 13636-13652. https://doi.org/10.1109/tpami.2023.3296823

Vancouver

Deng J, Yang Z, Liu D, Chen T, Zhou W, Zhang Y, et al. TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(11):13636-13652. doi:10.1109/tpami.2023.3296823

BibTeX
@article{Deng2023,
  author = {Deng, Jiajun and Yang, Zhengyuan and Liu, Daqing and Chen, Tianlang and Zhou, Wengang and Zhang, Yanyong and Li, Houqiang and Ouyang, Wanli},
  title = {TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {11},
  pages = {13636--13652},
  doi = {10.1109/tpami.2023.3296823}
}
Kostenlos starten →