EUVIMEDEUVIMED entdecken
Crossref

Exploring Vision-Language Foundation Model for Novel Object Captioning

Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, Tao Mei

IEEE Transactions on Circuits and Systems for Video Technology · 2025-01 · 35 (1) · 91-102

DOI: 10.1109/tcsvt.2024.3452437 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-01. Verfasst von Jianjie Luo und 6 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 57 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

It is always well believed that pre-trained vision-language foundation models (e.g., CLIP) would substantially facilitate vision-language tasks. Nevertheless, there has been less evidence in support of the idea on describing novel objects in images. In this paper, we propose the Novel Object Transformer with CLIP (NOTC), a Transformer-based model that innovatively exploits the powerful vision-language representation ability of CLIP to enhance novel object captioning model’s training and sentence decoding processes. Technically, given the primary bag-of-objects extracted by Faster R-CNN, NOTC first capitalize on an object distiller module to emphasize the most salient objects and infer the missing novel ones. The refined object words are additionally fed into the object-centric word predictor to generate sentence word-by-word. During training, we design a CLIP-based self-critical sequence training paradigm to select visually-grounded sampled sentence with higher CLIP score reward, which enables a joint training process of captioning model over out-domain training images with novel objects. Moreover, at inference, a new CLIP beam search algorithm is devised to enforce the existence of novel objects and encourage the partial word sequences with higher CLIP scores, thereby decoding both visually-grounded and comprehensive sentences. Extensive experiments are conducted on held-out COCO and nocaps datasets, and competitive performances are reported when compared to state-of-the-art approaches.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., & Mei, T. (2025). Exploring Vision-Language Foundation Model for Novel Object Captioning. IEEE Transactions on Circuits and Systems for Video Technology, 35(1), 91-102. https://doi.org/10.1109/tcsvt.2024.3452437

Vancouver

Luo J, Li Y, Pan Y, Yao T, Feng J, Chao H, et al. Exploring Vision-Language Foundation Model for Novel Object Captioning. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(1):91-102. doi:10.1109/tcsvt.2024.3452437

BibTeX
@article{Luo2025,
  author = {Luo, Jianjie and Li, Yehao and Pan, Yingwei and Yao, Ting and Feng, Jianlin and Chao, Hongyang and Mei, Tao},
  title = {Exploring Vision-Language Foundation Model for Novel Object Captioning},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  volume = {35},
  number = {1},
  pages = {91--102},
  doi = {10.1109/tcsvt.2024.3452437}
}
Kostenlos starten →