Visual-Language Scene-Relation-Aware Zero-Shot Captioner
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-10 · 47 (10) · 8725-8739
DOI: 10.1109/tpami.2025.3581174 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-10. Verfasst von Qianyue Bao und 7 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 104 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Joint Fund Project of National Natural Science Foundation of China, Postdoctoral Fellowship Program of China Postdoctoral Science Foundation.
Abstract (Originalsprache)
Zero-shot image captioning can harness the knowledge of pre-trained visual language models (VLMs) and language models (LMs) to generate captions for target domain images without paired sample training. Existing methods attempt to establish high-quality connections between visual and textual modalities in text-only pre-training tasks. These methods can be divided into two perspectives: sentence-level and entity-level. Although they achieve effective performance on some metrics, they suffer from hallucinations due to biased associations during training. In this paper, we propose a scene-relation-level pre-training task by considering relations as more valuable modal connection bridges. Based on this, we construct a novel Visual-Language Scene Relation Aware Captioner (SRACap), which expands the ability to predict scene relations while generating captions for images. In addition, SRACap possesses excellent cross-domain zero-shot generalization capability, which is driven by a well-designed scene reinforcement switching pipeline. We introduce a scene policy network to dynamically crop salient regions from images and feed them into a language model to generate captions. We integrate multiple expert CLIP models to form a mixture-of-rewards module (MoR) as a reward source, and deeply optimized SRACap through the policy gradient algorithm in the zero-shot inference stage. With the iteration of scene reinforcement switching, SRACap can gradually refine the generated caption details while maintaining high semantic consistency across visual-linguistic modalities. We conduct extensive experiments on multiple standard image captioning benchmarks, showing that SRACap can accurately understand scene structures and generate high-quality text, significantly outperforming other zero-shot inference methods.
Förderung
- National Natural Science Foundation of China
- Joint Fund Project of National Natural Science Foundation of China
- Postdoctoral Fellowship Program of China Postdoctoral Science Foundation
- Fund for Foreign Scholars in University Research and Teaching Programs
- Program for Cheung Kong Scholars and Innovative Research Team in University
- Key Scientific Technological Innovation Research Project by the Ministry of Education
- Fundamental Research Funds for the Central Universities
- Innovation Fund of Xidian University
Zitieren
APA
Bao, Q., Liu, F., Jiao, L., Liu, Y., Li, S., Li, L., Liu, X., & Chen, P. (2025). Visual-Language Scene-Relation-Aware Zero-Shot Captioner. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(10), 8725-8739. https://doi.org/10.1109/tpami.2025.3581174
Vancouver
Bao Q, Liu F, Jiao L, Liu Y, Li S, Li L, et al. Visual-Language Scene-Relation-Aware Zero-Shot Captioner. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(10):8725-8739. doi:10.1109/tpami.2025.3581174
BibTeX
@article{Bao2025,
author = {Bao, Qianyue and Liu, Fang and Jiao, Licheng and Liu, Yang and Li, Shuo and Li, Lingling and Liu, Xu and Chen, Puhua},
title = {Visual-Language Scene-Relation-Aware Zero-Shot Captioner},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {10},
pages = {8725--8739},
doi = {10.1109/tpami.2025.3581174}
}
EUVIMED