EUVIMEDEUVIMED entdecken
Crossref

Instruction-Guided Scene Text Recognition

Yongkun Du, Zhineng Chen, Yuchen Su, Caiyan Jia, Yu-Gang Jiang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-04 · 47 (4) · 2723-2738

DOI: 10.1109/tpami.2025.3525526 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-04. Verfasst von Yongkun Du und 4 weiteren Autor:innen. Die Arbeit wurde bisher 43-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multi-modal models have shown appealing performance in visual recognition tasks, as free-form text-guided training evokes the ability to understand fine-grained visual content. However, current models cannot be trivially applied to scene text recognition (STR) due to the compositional difference between natural and text images. We propose a novel instruction-guided scene text recognition (IGTR) paradigm that formulates STR as an instruction learning problem and understands text images by predicting character attributes, e.g., character frequency, position, etc. IGTR first devises instruction triplets, providing rich and diverse descriptions of character attributes. To effectively learn these attributes through question-answering, IGTR develops a lightweight instruction encoder, a cross-modal feature fusion module and a multi-task answer head, which guides nuanced text image understanding. Furthermore, IGTR realizes different recognition pipelines simply by using different instructions, enabling a character-understanding-based text reasoning paradigm that differs from current methods considerably. Experiments on English and Chinese benchmarks show that IGTR outperforms existing models by significant margins, while maintaining a small model size and fast inference speed. Moreover, by adjusting the sampling of instructions, IGTR offers an elegant way to tackle the recognition of rarely appearing and morphologically similar characters, which were previous challenges.

Förderung

  • National Key R&D Program of China
  • National Natural Science Foundation of China

Zitieren

APA

Du, Y., Chen, Z., Su, Y., Jia, C., & Jiang, Y. G. (2025). Instruction-Guided Scene Text Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4), 2723-2738. https://doi.org/10.1109/tpami.2025.3525526

Vancouver

Du Y, Chen Z, Su Y, Jia C, Jiang YG. Instruction-Guided Scene Text Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(4):2723-2738. doi:10.1109/tpami.2025.3525526

BibTeX
@article{Du2025,
  author = {Du, Yongkun and Chen, Zhineng and Su, Yuchen and Jia, Caiyan and Jiang, Yu-Gang},
  title = {Instruction-Guided Scene Text Recognition},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {4},
  pages = {2723--2738},
  doi = {10.1109/tpami.2025.3525526}
}
Kostenlos starten →