EUVIMEDEUVIMED entdecken
Crossref

Understand Layout and Translate Text: Unified Feature-Conductive End-to-End Document Image Translation

Zhiyang Zhang, Yaping Zhang, Yupu Liang, Cong Ma, Lu Xiang, Yang Zhao, Yu Zhou, Chengqing Zong

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-05 · 47 (5) · 3358-3376

DOI: 10.1109/tpami.2025.3530998 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-05. Verfasst von Zhiyang Zhang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 71 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Document Image Translation (DIT) aims to translate texts on document images from one language to another. It is a multi-modal task involving cooperation of text and layout. Current approaches either handle layout and translation as separate processes, risking accumulative errors, or use vanilla end-to-end encoder-decoder models to capture layout implicitly, often suffering inadequate layout incorporation. We argue that a favorable framework should explicitly engage layout-specific modules and properly organize them toward translation. For this, we first revisit two key layouts: the geometric layout reflecting word's spatial positions, and the logical layout depicting word's logical order. Then, a novel pipeline (understand layout $\rightarrow$→ translate text) is determined to prioritize layouts such that preceding layouts contribute to translation. Following this pipeline, we introduce Unified Document Image Translation (UniDIT), a comprehensive framework that unifies layout with translation in one network. It is devised to leverage each module's advantage, and provide an elaborate feature-conductive flow for module communication globally. A novel bridging mechanism is also introduced to adapt layout features conducive to translation. We further contribute DITransv2, a large-scale fine-grained benchmark that includes heterogeneous and complex document layouts. Extensive experiments on DITransv2 and additional established benchmarks demonstrate UniDIT outperforms previous state-of-the-arts in all aspects.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Zhang, Z., Zhang, Y., Liang, Y., Ma, C., Xiang, L., Zhao, Y., Zhou, Y., & Zong, C. (2025). Understand Layout and Translate Text: Unified Feature-Conductive End-to-End Document Image Translation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5), 3358-3376. https://doi.org/10.1109/tpami.2025.3530998

Vancouver

Zhang Z, Zhang Y, Liang Y, Ma C, Xiang L, Zhao Y, et al. Understand Layout and Translate Text: Unified Feature-Conductive End-to-End Document Image Translation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(5):3358-3376. doi:10.1109/tpami.2025.3530998

BibTeX
@article{Zhang2025,
  author = {Zhang, Zhiyang and Zhang, Yaping and Liang, Yupu and Ma, Cong and Xiang, Lu and Zhao, Yang and Zhou, Yu and Zong, Chengqing},
  title = {Understand Layout and Translate Text: Unified Feature-Conductive End-to-End Document Image Translation},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {5},
  pages = {3358--3376},
  doi = {10.1109/tpami.2025.3530998}
}
Kostenlos starten →