TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-05 · 48 (5) · 6008-6019
DOI: 10.1109/tpami.2026.3653415 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-05. Verfasst von Yuliang Liu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 37-mal zitiert (Crossref) und nennt 69 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
We present TextMonkey, a large multimodal model (LMM) tailored for text-centric tasks. Our approach introduces enhancement across several dimensions: By adopting Shifted Window Attention layer, we achieve cross-window connectivity at higher input resolutions and stabilize early training; We hypothesize that images may contain redundant tokens, and by using similarity to filter out significant tokens, we can not only streamline the token length but also enhance the model's performance. Moreover, by expanding our model's capabilities to encompass text spotting and grounding, and incorporating positional information into responses, we enhance interpretability. Evaluation on 12 benchmarks shows notable improvements: 5.2% in Scene Text-Centric tasks (including STVQA, TextVQA, and OCRVQA), 6.9% in Document-Oriented tasks (such as DocVQA, InfoVQA, ChartVQA, DeepForm, Kleister Charity, and WikiTableQuestions), and 2.8% in Key Information Extraction tasks (comprising FUNSD, SROIE, and POIE). It outperforms in scene text spotting with a 10.9% increase and sets a new standard on OCRBench, a comprehensive benchmark consisting of 29 OCR-related assessments, with a score of 561, surpassing previous open-sourced large multimodal models for document understanding.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Liu, Y., Yang, B., Liu, Q., Li, Z., Ma, Z., Zhang, S., & Bai, X. (2026). TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(5), 6008-6019. https://doi.org/10.1109/tpami.2026.3653415
Vancouver
Liu Y, Yang B, Liu Q, Li Z, Ma Z, Zhang S, et al. TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(5):6008-6019. doi:10.1109/tpami.2026.3653415
BibTeX
@article{Liu2026,
author = {Liu, Yuliang and Yang, Biao and Liu, Qiang and Li, Zhang and Ma, Zhiyin and Zhang, Shuo and Bai, Xiang},
title = {TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
volume = {48},
number = {5},
pages = {6008--6019},
doi = {10.1109/tpami.2026.3653415}
}
EUVIMED