EUVIMEDEUVIMED entdecken
Crossref

Visual-Language Scene-Relation-Aware Zero-Shot Captioner

Qianyue Bao, Fang Liu, Licheng Jiao, Yang Liu, Shuo Li, Lingling Li, Xu Liu, Puhua Chen

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-10 · 47 (10) · 8725-8739

DOI: 10.1109/tpami.2025.3581174 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-10. Verfasst von Qianyue Bao und 7 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 104 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Joint Fund Project of National Natural Science Foundation of China, Postdoctoral Fellowship Program of China Postdoctoral Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Zero-shot image captioning can harness the knowledge of pre-trained visual language models (VLMs) and language models (LMs) to generate captions for target domain images without paired sample training. Existing methods attempt to establish high-quality connections between visual and textual modalities in text-only pre-training tasks. These methods can be divided into two perspectives: sentence-level and entity-level. Although they achieve effective performance on some metrics, they suffer from hallucinations due to biased associations during training. In this paper, we propose a scene-relation-level pre-training task by considering relations as more valuable modal connection bridges. Based on this, we construct a novel Visual-Language Scene Relation Aware Captioner (SRACap), which expands the ability to predict scene relations while generating captions for images. In addition, SRACap possesses excellent cross-domain zero-shot generalization capability, which is driven by a well-designed scene reinforcement switching pipeline. We introduce a scene policy network to dynamically crop salient regions from images and feed them into a language model to generate captions. We integrate multiple expert CLIP models to form a mixture-of-rewards module (MoR) as a reward source, and deeply optimized SRACap through the policy gradient algorithm in the zero-shot inference stage. With the iteration of scene reinforcement switching, SRACap can gradually refine the generated caption details while maintaining high semantic consistency across visual-linguistic modalities. We conduct extensive experiments on multiple standard image captioning benchmarks, showing that SRACap can accurately understand scene structures and generate high-quality text, significantly outperforming other zero-shot inference methods.

Förderung

  • National Natural Science Foundation of China
  • Joint Fund Project of National Natural Science Foundation of China
  • Postdoctoral Fellowship Program of China Postdoctoral Science Foundation
  • Fund for Foreign Scholars in University Research and Teaching Programs
  • Program for Cheung Kong Scholars and Innovative Research Team in University
  • Key Scientific Technological Innovation Research Project by the Ministry of Education
  • Fundamental Research Funds for the Central Universities
  • Innovation Fund of Xidian University

Zitieren

APA

Bao, Q., Liu, F., Jiao, L., Liu, Y., Li, S., Li, L., Liu, X., & Chen, P. (2025). Visual-Language Scene-Relation-Aware Zero-Shot Captioner. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(10), 8725-8739. https://doi.org/10.1109/tpami.2025.3581174

Vancouver

Bao Q, Liu F, Jiao L, Liu Y, Li S, Li L, et al. Visual-Language Scene-Relation-Aware Zero-Shot Captioner. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(10):8725-8739. doi:10.1109/tpami.2025.3581174

BibTeX
@article{Bao2025,
  author = {Bao, Qianyue and Liu, Fang and Jiao, Licheng and Liu, Yang and Li, Shuo and Li, Lingling and Liu, Xu and Chen, Puhua},
  title = {Visual-Language Scene-Relation-Aware Zero-Shot Captioner},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {10},
  pages = {8725--8739},
  doi = {10.1109/tpami.2025.3581174}
}
Kostenlos starten →