Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection
IEEE Transactions on Image Processing · 2024 · 33 · 6253-6267
DOI: 10.1109/tip.2024.3485518 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2024. Verfasst von Yifan Xu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 90 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Pengcheng Laboratory Research Project, National Science and Technology Major Project.
Abstract (Originalsprache)
We explore multi-modal contextual knowledge learned through multi-modal masked language modeling to provide explicit localization guidance for novel classes in open-vocabulary object detection (OVD). Intuitively, a well-modeled and correctly predicted masked concept word should effectively capture the textual contexts, visual contexts, and the cross-modal correspondence between texts and regions, thereby automatically activating high attention on corresponding regions. In light of this, we propose a multi-modal contextual knowledge distillation framework, MMC-Det, to explicitly supervise a student detector with the context-aware attention of the masked concept words in a teacher fusion transformer. The teacher fusion transformer is trained with our newly proposed diverse multi-modal masked language modeling (D-MLM) strategy, which significantly enhances the fine-grained region-level visual context modeling in the fusion transformer. The proposed distillation process provides additional contextual guidance to the concept-region matching of the detector, thereby further improving the OVD performance. Extensive experiments performed upon various detection datasets show the effectiveness of our multi-modal context learning strategy.
Förderung
- National Natural Science Foundation of China
- Pengcheng Laboratory Research Project
- National Science and Technology Major Project
- Alibaba Innovative Research Program; and in part by the CAS Project for Young Scientists in Basic Research
Zitieren
APA
Xu, Y., Zhang, M., Yang, X., & Xu, C. (2024). Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection. IEEE Transactions on Image Processing, 33, 6253-6267. https://doi.org/10.1109/tip.2024.3485518
Vancouver
Xu Y, Zhang M, Yang X, Xu C. Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection. IEEE Transactions on Image Processing. 2024;33:6253-6267. doi:10.1109/tip.2024.3485518
BibTeX
@article{Xu2024,
author = {Xu, Yifan and Zhang, Mengdan and Yang, Xiaoshan and Xu, Changsheng},
title = {Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection},
journal = {IEEE Transactions on Image Processing},
year = {2024},
volume = {33},
pages = {6253--6267},
doi = {10.1109/tip.2024.3485518}
}
EUVIMED