Fine-Grained Image Captioning by Ranking Diffusion Transformer
IEEE Transactions on Image Processing · 2025 · 34 · 8332-8344
DOI: 10.1109/tip.2025.3641303 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Jun Wan und 6 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Natural Science Foundation of Hubei Province.
Abstract (Originalsprache)
The CLIP visual feature-based image captioning models have developed rapidly and achieved remarkable results. However, existing models still struggle to produce descriptive and discriminative captions because they insufficiently exploit fine-grained visual cues and fail to model complex vision-language alignment. To address these limitations, we propose a Ranking Diffusion Transformer (RDT), which integrates a Ranking Visual Encoder (RVE) and a Ranking Loss (RL) for fine-grained image captioning. The RVE introduces a novel ranking attention mechanism that effectively mines diverse and discriminative visual information from CLIP features. Meanwhile, the RL leverages the ranking of generated caption quality as a global semantic supervisory signal, thereby enhancing the diffusion process and strengthening vision-language semantic alignment. We show that by collaborating RVE and RL via the novel RDT-and by gradually adding and removing noise in the diffusion process-more discriminative visual features are learned and precisely aligned with the language features. Experimental results on popular benchmark datasets demonstrate that our proposed RDT surpasses existing state-of-the-art image captioning models in the literature. The code is publicly available at: https://github.com/junwan2014/RDT.
Förderung
- National Natural Science Foundation of China
- Natural Science Foundation of Hubei Province
Zitieren
APA
Wan, J., Gan, M., Zhang, L., Zhou, J., Liu, J., Du, B., & Chen, C. L. P. (2025). Fine-Grained Image Captioning by Ranking Diffusion Transformer. IEEE Transactions on Image Processing, 34, 8332-8344. https://doi.org/10.1109/tip.2025.3641303
Vancouver
Wan J, Gan M, Zhang L, Zhou J, Liu J, Du B, et al. Fine-Grained Image Captioning by Ranking Diffusion Transformer. IEEE Transactions on Image Processing. 2025;34:8332-8344. doi:10.1109/tip.2025.3641303
BibTeX
@article{Wan2025,
author = {Wan, Jun and Gan, Min and Zhang, Lefei and Zhou, Jie and Liu, Jun and Du, Bo and Chen, C L Philip},
title = {Fine-Grained Image Captioning by Ranking Diffusion Transformer},
journal = {IEEE Transactions on Image Processing},
year = {2025},
volume = {34},
pages = {8332--8344},
doi = {10.1109/tip.2025.3641303}
}
EUVIMED