EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Image Captioning via Compact Bidirectional Architecture

Zijie Song, Yuanen Zhou, Zhenzhen Hu, Daqing Liu, Huixia Ben, Richang Hong, Meng Wang

IEEE Transactions on Multimedia · 2026 · 1-12

DOI: 10.1109/tmm.2026.3680397 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Multimedia, 2026. Verfasst von Zijie Song und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Most current image captioning models typically generate captions from left-to-right. This unidirectional property makes them can only leverage past context but not future context. Though refinement-based models can exploit both past and future context by generating a new caption in the second stage based on pre-retrieved or pre-generated captions in the first stage, the decoder of these models generally consists of two networks~(i.e. a retriever or captioner in the first stage and a captioner in the second stage), which can only be executed sequentially. In this paper, we introduce a Compact Bidirectional Transformer model for image captioning that can leverage bidirectional context implicitly and explicitly while the decoder can be executed parallelly. Specifically, it is implemented by tightly coupling left-to-right(L2R) and right-to-left(R2L) flows into a single compact model to serve as a regularization for implicitly exploiting bidirectional context and optionally allowing explicit interaction of the bidirectional flows, while the final caption is chosen from either L2R or R2L flow in a sentence-level ensemble manner. We conduct extensive ablation studies on MSCOCO benchmark and find that the compact bidirectional architecture and the sentence-level ensemble play more important roles than the explicit interaction mechanism. By combining with word-level ensemble seamlessly, the effect of sentence-level ensemble is further enlarged. We further extend the conventional one-flow self-critical training to the two-flows version under this architecture and achieve new state-of-the-art results in comparison with non-vision-language-pretraining models. Finally, we verify the generality of this compact bidirectional architecture by extending it to LSTM backbone. Source code is available at https://github.com/YuanEZhou/cbtic.

Zitieren

APA

Song, Z., Zhou, Y., Hu, Z., Liu, D., Ben, H., Hong, R., & Wang, M. (2026). Image Captioning via Compact Bidirectional Architecture. IEEE Transactions on Multimedia, 1-12. https://doi.org/10.1109/tmm.2026.3680397

Vancouver

Song Z, Zhou Y, Hu Z, Liu D, Ben H, Hong R, et al. Image Captioning via Compact Bidirectional Architecture. IEEE Transactions on Multimedia. 2026:1-12. doi:10.1109/tmm.2026.3680397

BibTeX
@article{Song2026,
  author = {Song, Zijie and Zhou, Yuanen and Hu, Zhenzhen and Liu, Daqing and Ben, Huixia and Hong, Richang and Wang, Meng},
  title = {Image Captioning via Compact Bidirectional Architecture},
  journal = {IEEE Transactions on Multimedia},
  year = {2026},
  pages = {1--12},
  doi = {10.1109/tmm.2026.3680397}
}
Kostenlos starten →