EUVIMEDEUVIMED entdecken
Crossref

DSTA: Reinforcing Vision-Language Understanding for Scene-Text VQA With Dual-Stream Training Approach

Yingtao Tan, Yingying Chen, Jinqiao Wang

IEEE Signal Processing Letters · 2025 · 32 · 6-10

DOI: 10.1109/lsp.2024.3486104 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Signal Processing Letters, 2025. Verfasst von Yingtao Tan und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 35 Literaturangaben. Gefördert von: Strategic Priority Research Program of the Chinese Academy of Sciences.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Scene-Text Visual Question Answering (STVQA) is a comprehensive task that requires reading and understanding the text in images to answer the question. Existing methods of exploring the vision-language relationships between questions, images, and scene text have achieved impressive results. However, these studies heavily rely on auxiliary modules, such as external OCR systems and object detection networks, making the question-answering process cumbersome and highly dependent. In addition, OCR text is treated as textual content only in these approaches, while its visual learning is ignored. To alleviate the above problems, we propose a novel end-to-end dual-stream multi-loss training approach called DSTA. Our model first integrates a text spotter into multimodal learning to incorporate overall textual and visual OCR features. Specifically, we propose a novel dual-stream multi-loss training strategy that improves multimodal understanding while training question-answering. In addition, we design OCR Contrastive Learning (OCL) to enhance vision-language understanding by exploring the multimodal features of OCR text in depth. Experiments show that DSTA outperforms previous state-of-the-art methods on two STVQA benchmarks without any extra training data.

Förderung

  • Strategic Priority Research Program of the Chinese Academy of Sciences

Zitieren

APA

Tan, Y., Chen, Y., & Wang, J. (2025). DSTA: Reinforcing Vision-Language Understanding for Scene-Text VQA With Dual-Stream Training Approach. IEEE Signal Processing Letters, 32, 6-10. https://doi.org/10.1109/lsp.2024.3486104

Vancouver

Tan Y, Chen Y, Wang J. DSTA: Reinforcing Vision-Language Understanding for Scene-Text VQA With Dual-Stream Training Approach. IEEE Signal Processing Letters. 2025;32:6-10. doi:10.1109/lsp.2024.3486104

BibTeX
@article{Tan2025,
  author = {Tan, Yingtao and Chen, Yingying and Wang, Jinqiao},
  title = {DSTA: Reinforcing Vision-Language Understanding for Scene-Text VQA With Dual-Stream Training Approach},
  journal = {IEEE Signal Processing Letters},
  year = {2025},
  volume = {32},
  pages = {6--10},
  doi = {10.1109/lsp.2024.3486104}
}
Kostenlos starten →