Instruction-Guided Scene Text Recognition
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-04 · 47 (4) · 2723-2738
DOI: 10.1109/tpami.2025.3525526 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-04. Verfasst von Yongkun Du und 4 weiteren Autor:innen. Die Arbeit wurde bisher 43-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China.
Abstract (Originalsprache)
Multi-modal models have shown appealing performance in visual recognition tasks, as free-form text-guided training evokes the ability to understand fine-grained visual content. However, current models cannot be trivially applied to scene text recognition (STR) due to the compositional difference between natural and text images. We propose a novel instruction-guided scene text recognition (IGTR) paradigm that formulates STR as an instruction learning problem and understands text images by predicting character attributes, e.g., character frequency, position, etc. IGTR first devises instruction triplets, providing rich and diverse descriptions of character attributes. To effectively learn these attributes through question-answering, IGTR develops a lightweight instruction encoder, a cross-modal feature fusion module and a multi-task answer head, which guides nuanced text image understanding. Furthermore, IGTR realizes different recognition pipelines simply by using different instructions, enabling a character-understanding-based text reasoning paradigm that differs from current methods considerably. Experiments on English and Chinese benchmarks show that IGTR outperforms existing models by significant margins, while maintaining a small model size and fast inference speed. Moreover, by adjusting the sampling of instructions, IGTR offers an elegant way to tackle the recognition of rarely appearing and morphologically similar characters, which were previous challenges.
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
Zitieren
APA
Du, Y., Chen, Z., Su, Y., Jia, C., & Jiang, Y. G. (2025). Instruction-Guided Scene Text Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4), 2723-2738. https://doi.org/10.1109/tpami.2025.3525526
Vancouver
Du Y, Chen Z, Su Y, Jia C, Jiang YG. Instruction-Guided Scene Text Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(4):2723-2738. doi:10.1109/tpami.2025.3525526
BibTeX
@article{Du2025,
author = {Du, Yongkun and Chen, Zhineng and Su, Yuchen and Jia, Caiyan and Jiang, Yu-Gang},
title = {Instruction-Guided Scene Text Recognition},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {4},
pages = {2723--2738},
doi = {10.1109/tpami.2025.3525526}
}
EUVIMED